I run a CentOS 5.7 64 machine with 24gb ram and running kernel 2.6.18-274.12.1.el5.

This machine runs only Nginx, php-fpm and Xcache as extra applications.

Since about 3 weeks my memory behavior on this machine has changed and I cannot explain why. There are no crons running which flush anything like this. There are also no large numbers of files being deleted/changed during these drops.

The 'cached' memory gets dropped about every few hours, but it's never a set gap between flushes, this indicates to me that some bottleneck gets reached instead. It also always seems to be when total memory usages gets to about 18GB, but again, not always exactly 18GB.

This is a graph of my memory usage: enter image description here

As you can see in the graph the 'buffers' always stay more or less the same, it is mainly the 'cache' that gets dropped.

Running vmstat -m I have outputted the memory usage just before and just after a memory drop. The output is here: http://pastebin.com/diff.php?i=hJqZqztm 'old version' being before, 'new version' being after a drop.

About 3 weeks ago my server crashed during a heavy DDOS attack, after I rebooted the machine this odd behavior started. I have checked a bunch of logs, restarted the machine again, and cannot find any indication what changed.

During these 'cache' memory drops, my iNode usage drops at the same time.

enter image description here

Does anyone have any idea what might be causing this behavior? Clearly my RAM isn't full, so I am curious why this could be happening.

  • Your 'buffers' value is really really high. That metric (although commonly misinterprested as dirty writes) is in effect block-device cache (if you cat /dev/sda that value will go up). It also could go up if you have lots of filesystems or LVM/Raid going on. But, 9G is enormous. What type of media setup do you use? FYI on most servers I rarely see that value go over 1G, let alone 9G. – Matthew Ife Jan 22 '12 at 17:40
  • This server hosts over 25 million text files (usually a only a few kb each, spread over 4x sas 15k rpm drives in raid10 setup), those files are spread in a /25/25/25/25/ folder structure. And these files are requested via PHP script quite often. 2 files are requested per page, and about 30 pages per second. As far as I understand Linux caches the most accessed files in the RAM. Storing these files a MySQL DB was no option as the DB size would be too big. But nothing has changes in the structure since the last 3 weeks, so the change in memory behavior is a mystery to us. – Mr.Boon Jan 22 '12 at 18:35
  • Does your disk I/O correllate with this behaviour? – Matthew Ife Jan 22 '12 at 18:48
  • Yes it does. When 'cache' is flushed, more info will need to come directly from disk, so there are 'read' spikes. You can see it here: i.imgur.com/3mmDi.png. This shouldn't have to happen if the server wouldn't flush that cache. – Mr.Boon Jan 22 '12 at 18:51
  • I have to admit the graph confuses me (I dont use munin). The text stats indicate a min/max of buffers to be 8.87G to 9.04G respectively, but the graph displays a number of spikes between 13G and 18G. Perhaps I am reading the legend wrong here? And, which is actually right, the graph or the text summary.. I guess the graph on the basis of what your I/O chart shows. – Matthew Ife Jan 22 '12 at 19:01

What does a jagged committed memory graph mean?

Sockets create inodes when a connection is accept()-ed, so the inode behaviour could be from a massive burst of connections being opened or closed, respectively. This could happen when (as in the linked question) logrotate kills a bunch of FastCGI processes. Not sure if this would apply to php-fpm.

Just a wild theory, and it doesn't really explain why the cache is cleared at the same time. Still, could be worth a look?


solved it by putting vm.zone_reclaim_mode = 0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.