2

I run a CentOS 5.6 (64bit) machine that has Nginx (latest version) running, with php-fpm (latest version). Things run very well, but since about 2 weeks I noticed in my Munin graphs that about every 2 hours the 'cache' usages drops. Before it used be a steady fully graph, that didn't seem to reset every so often.

PHP-FPM settings:

pm.max_children = 300
daemonize = yes
pm = static
listen = /tmp/fpm.sock
pm.max_requests = 1000

I have checked the php-fpm.log, and about once per 5 seconds a child process is killed, and restarted. But this is all the time, so this does not explain the sudden drops.

munin graph

I only run Nginx, PHP (via fpm), Munin and vsftpd on this machine. No crons run at exactly the time of the drops.

My question: What could be causing these drops in cache usage?

Improve this question
3
  • Check your cron jobs for anything that runs at those times.
    – David Schwartz
    Jan 18, 2012 at 20:59
  • I have, but as you can see on the graphs the drops aren't at the same time every time. But I do not have a cron running that could cause such a drop. Only simple PHP scripts that take less than a second to run, and perform simple queries.
    – Mr.Boon
    Jan 19, 2012 at 8:31
  • I noticed that the disk IO graphs also spike up a little at exactly the points when the memory graphs dip down. This make sense of course as information that was once served by the ram, now has to be read from the disks again.
    – Mr.Boon
    Jan 19, 2012 at 14:31

3 Answers 3

Reset to default
0

I'm not sure about CentOS 5.6, but for my Linux 2.6.32-5-amd64 adding this line in /etc/sysctl.conf

vm.vfs_cache_pressure = 1

resolves this problem.

Improve this answer
1
  • Could you explain this answer a little bit please? This value is currently on 100.
    – Mr.Boon
    Feb 6, 2012 at 18:51
0

I think some large file that's part of your webserver is responsible. I'm guessing a log that is rotated or compressed as a function of general housekeeping. Below are some experiments showing that Linux dumps that part of the cache when a file is removed, including by means of a utility like gzip.

$ free
             total       used       free     shared    buffers     cached
Mem:       4054352    3867292     187060          0      75664     390788
-/+ buffers/cache:    3400840     653512
Swap:      4194300     338344    3855956

# echo 1 > /proc/sys/vm/drop_caches
$ free
             total       used       free     shared    buffers     cached
Mem:       4054352    3420612     633740          0        292      20516
-/+ buffers/cache:    3399804     654548
Swap:      4194300     338344    3855956

But that's probably not what's happening. What if a file were deleted? Would it still be referenced?

$ dd if=/dev/zero of=cachetest bs=1M count=200
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 0.288883 s, 726 MB/s

$ free
             total       used       free     shared    buffers     cached
Mem:       4054352    3647688     406664          0       3200     240328
-/+ buffers/cache:    3404160     650192
Swap:      4194300     338344    3855956

$ rm cachetest
$ free

             total       used       free     shared    buffers     cached
Mem:       4054352    3438508     615844          0       3208      35528
-/+ buffers/cache:    3399772     654580
Swap:      4194300     338344    3855956

And, if we reload it then compress the file:

$gzip cachetest
$free
             total       used       free     shared    buffers     cached
Mem:       4054352    3439412     614940          0       3384      36404
-/+ buffers/cache:    3399624     654728
Swap:      4194300     338344    3855956
Improve this answer
3
  • Thank you for your answer. Here is another longer munin graph, i.imgur.com/Vtauz.png there you can see how it used to be fine. Then all the sudden it started doing these strange flushes. It seems to me that it is hitting some kind of limit, once that is reached it flushes the cache. Why this happens I have no clue.
    – Mr.Boon
    Jan 19, 2012 at 17:06
  • @Boon Config, software, kernel version possibly updated during a reboot?
    – Jeff Ferland
    Jan 19, 2012 at 17:23
  • The server was indeed restarted. PHP, Nginx didn't update, maybe something else, but I wouldn't know what that could be?
    – Mr.Boon
    Jan 19, 2012 at 17:35
0

Solved it by putting vm.zone_reclaim_mode = 0

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .