I have a server where I run worker processes that mmap serveral rather big read-only "dictionary" files (~8GB total). Tests showed that they actively access only around ~1GB worth of the pages in these files. On the same server I run another process that sequentially reads a huge file, merges some updates to it and writes the result in a new version of this huge file. There in no other major activity on the server besides this "merger" process and "worker" processes. So normally the workers should be CPU-bound and the merger should be disk-bound. But what I see is that the workers are constantly choking on major page faults. The merger uses around 20GB of RSS and the machine has 48GB. There are 4 workers. They have 2GB RSS each and only 600MB shared (instead of expected 1GB of hot pages). Somehow rest of the memory is mostly used by fs cache. Is there a way to "prioritize" the hot pages of my mmaped files into memory? I tried madvise(MADV_WILLNEED) but it doesn't seem to help. Maybe there is a solution with cgroups or sysctls?
$ free total used free shared buffers cached Mem: 49324064 48863392 460672 0 22520 25409896 -/+ buffers/cache: 23430976 25893088 Swap: 0 0 0 $ uname -a Linux dev-kiwi02 3.2.0-25-server #40-Ubuntu SMP Fri May 25 13:12:35 UTC 2012 x86_64 GNU/Linux
P.S. Asked this on StackOverflow already but looks like ServerFault is more appropriate.