Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I run a CentOS 5.6 (64bit) machine that has Nginx (latest version) running, with php-fpm (latest version). Things run very well, but since about 2 weeks I noticed in my Munin graphs that about every 2 hours the 'cache' usages drops. Before it used be a steady fully graph, that didn't seem to reset every so often.

PHP-FPM settings:

pm.max_children = 300
daemonize = yes
pm = static
listen = /tmp/fpm.sock
pm.max_requests = 1000

I have checked the php-fpm.log, and about once per 5 seconds a child process is killed, and restarted. But this is all the time, so this does not explain the sudden drops.

munin graph

I only run Nginx, PHP (via fpm), Munin and vsftpd on this machine. No crons run at exactly the time of the drops.

My question: What could be causing these drops in cache usage?

share|improve this question
    
Check your cron jobs for anything that runs at those times. –  David Schwartz Jan 18 '12 at 20:59
    
I have, but as you can see on the graphs the drops aren't at the same time every time. But I do not have a cron running that could cause such a drop. Only simple PHP scripts that take less than a second to run, and perform simple queries. –  Mr.Boon Jan 19 '12 at 8:31
    
I noticed that the disk IO graphs also spike up a little at exactly the points when the memory graphs dip down. This make sense of course as information that was once served by the ram, now has to be read from the disks again. –  Mr.Boon Jan 19 '12 at 14:31

3 Answers 3

up vote 0 down vote accepted

I'm not sure about CentOS 5.6, but for my Linux 2.6.32-5-amd64 adding this line in /etc/sysctl.conf

vm.vfs_cache_pressure = 1

resolves this problem.

share|improve this answer
    
Could you explain this answer a little bit please? This value is currently on 100. –  Mr.Boon Feb 6 '12 at 18:51

I think some large file that's part of your webserver is responsible. I'm guessing a log that is rotated or compressed as a function of general housekeeping. Below are some experiments showing that Linux dumps that part of the cache when a file is removed, including by means of a utility like gzip.

$ free
             total       used       free     shared    buffers     cached
Mem:       4054352    3867292     187060          0      75664     390788
-/+ buffers/cache:    3400840     653512
Swap:      4194300     338344    3855956

# echo 1 > /proc/sys/vm/drop_caches
$ free
             total       used       free     shared    buffers     cached
Mem:       4054352    3420612     633740          0        292      20516
-/+ buffers/cache:    3399804     654548
Swap:      4194300     338344    3855956

But that's probably not what's happening. What if a file were deleted? Would it still be referenced?

$ dd if=/dev/zero of=cachetest bs=1M count=200
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 0.288883 s, 726 MB/s

$ free
             total       used       free     shared    buffers     cached
Mem:       4054352    3647688     406664          0       3200     240328
-/+ buffers/cache:    3404160     650192
Swap:      4194300     338344    3855956

$ rm cachetest
$ free

             total       used       free     shared    buffers     cached
Mem:       4054352    3438508     615844          0       3208      35528
-/+ buffers/cache:    3399772     654580
Swap:      4194300     338344    3855956

And, if we reload it then compress the file:

$gzip cachetest
$free
             total       used       free     shared    buffers     cached
Mem:       4054352    3439412     614940          0       3384      36404
-/+ buffers/cache:    3399624     654728
Swap:      4194300     338344    3855956
share|improve this answer
    
Thank you for your answer. Here is another longer munin graph, i.imgur.com/Vtauz.png there you can see how it used to be fine. Then all the sudden it started doing these strange flushes. It seems to me that it is hitting some kind of limit, once that is reached it flushes the cache. Why this happens I have no clue. –  Mr.Boon Jan 19 '12 at 17:06
    
@Boon Config, software, kernel version possibly updated during a reboot? –  Jeff Ferland Jan 19 '12 at 17:23
    
The server was indeed restarted. PHP, Nginx didn't update, maybe something else, but I wouldn't know what that could be? –  Mr.Boon Jan 19 '12 at 17:35

Solved it by putting vm.zone_reclaim_mode = 0

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.