Linux uses memory a little differently than you may expect. Why? It actually uses as much memory as it possibly can :)
The slowest point to any process, in any OS is waiting for I/O .. its not only slow, its downright dangerous at times. Imagine a process waiting in I/O bound calls that will never return .. and what happens if that process dies? That's why we have disk sleep to prevent kernel rot from within.
What remains is responsiveness be it server or desktop, the best compromise is to put frequently accessed files in memory so that processes do not become I/O bound. Its not at all unfair, in fact you can tweak how Linux uses cache vs swap, but 8/10 times keeping files in clean pages make things run much faster.
When a process skids to disk for memory (swap).. that's a bad sign. It happens often when you combine things that allocate much more memory than they actually need (per thread or fork) to deal with incoming requests of unknown size. It happens when running Apache + MySQL on the same computer, for instance. Other processes can't get contiguous blocks of memory from the kernel due to the two fat men standing in the middle of the doorway. When a process has to read a block device to access memory, everything gets slower, especially processes with clean allocated blocks waiting to write stuff to disk.
Some programmers use mlock() or even mlockall() to prevent this but usually the kernel knows better. Likewise, some people use O_DIRECT with open() to bypass buffering .. and unless you are writing a relational DB engine, the kernel usually knows better than you regarding what needs buffers or not. Actually, posix_fadvise() and posix_madvise() are better, as they suggest what you think you know to the kernel, rather than dictate. After all, your process is not the only thing running on any given computer :)
Probably a lot more information than you wanted. I meant to say, the usage looks absolutely normal. However, look at /proc/sys/vm for knobs to control this behavior, if you want to tinker :)