We are running a heavy service on Debian. Our service generates lots of file i/o and its performance relies on how well Linux kernel caches files.
Our system has 64GB memory. Total file size is 500GB (lots of small files). These files are served via reiserfs on raid. The kernel is 3.2.9 now. (this is not a new issue. We have this issue for several years.)
Usually, most of file content are cached and our system runs smoothly. free(1) command shows 14G memory cache, 10G memory buffer, and 8G memory are free.
However, in our peak hour, I found linux kernel released most of cache/buffer and kept them in free state. In my opinion, it is a huge waste to keeps memory in free state and I guess it is the reason that our service overloads in busy hours. free(1) command show something like 2G cache, 2G buffer, and 28G free.
My question is, in what situation linux kernel want to release memory instead of caching/buffering. And how to avoid this behavior? Is there any configuration we can tune?
(Let me know if you need more information about our hardware/system/service)