We are on a LAMP box. Centos.
Since yesterday we are experiencing a problem. Our httpd process count spikes to the maxclient limit every top of the hour. First the maxclient limit was 900. So it would spike up to 900.
Then we reduced it to 500. Now it spikes up to 500. Every hour in the very first minute of the hour sharp.
Since this happens in the first minute of the hour. We checked /var/log/cron log. There were a few jobs which ran close to the start of every hour. We turned those cron jobs off. Still no change. In the very first minute a spike starts. And lasts about 10-15 minutes.
Just as a side note. We are also seeing some dirtied inodes today. Also we have studied all recent deployments. We don't have any major changes.
Here is the server-status output just when the server starts hanging:
Current Time: Friday, 11-May-2012 21:02:42 UTC Restart Time: Friday, 11-May-2012 19:09:28 UTC Parent Server Generation: 0 Server uptime: 1 hour 53 minutes 13 seconds Total accesses: 1128389 - Total Traffic: 8.9 GB CPU Usage: u203.85 s36.51 cu.1 cs0 - 3.54% CPU load 166 requests/sec - 1.3 MB/second - 8.3 kB/request 500 requests currently being processed, 0 idle workers WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWKWWWWWWW WWWWWWWWWWWWWWRWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWKWWWWWWWWWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWCWWWWWWWWWWWWW WWWWWWWWWWWWWWWWWWWCWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW Scoreboard Key: "_" Waiting for Connection, "S" Starting up, "R" Reading Request, "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup, "C" Closing connection, "L" Logging, "G" Gracefully finishing, "I" Idle cleanup of worker, "." Open slot with no current process
/var/log/messages continously receives text as follows. Notice how some words are corrupt. I wonder if there is a system mechanism which tries to fix something every hour.:
May 11 22:46:43 www kernel: <ald(2825): WRITE block urnald(2825): WRITE block 128852880 on sdb1kjournald(2825): WRITE block 12885<7>kjournald(2825): WRITE block 128852896 on sdb1 May 11 22:46:43 www kernel: >kjournald(2825): WRITE block 128853 May 11 22:46:43 www kernel: WRITE block 128853104 on sdb5): WRITE block 128853112 on sdb1 May 11 22:46:43 www kernel: 2825): WRITE block 128853120 on sdb1 May 11 22:47:01 www kernel: ock 685806792 on sda1 May 11 22:47:05 www kernel: ock 129145376 on sdb1 May 11 22:47:10 www kernel: RITE block 789789472 on sda1 May 11 22:47:11 www kernel: ock 129189464 on sdb1 May 11 22:47:16 www kernel: <7ock 129360712 on sock 129360720 on sdb1 May 11 22:47:41 www kernel: <TE block 262819968 on sdb1
Where would you suggest I can track down the culprit which triggers this problem?