We are on a LAMP box. Centos.

Since yesterday we are experiencing a problem. Our httpd process count spikes to the maxclient limit every top of the hour. First the maxclient limit was 900. So it would spike up to 900.

Then we reduced it to 500. Now it spikes up to 500. Every hour in the very first minute of the hour sharp.

Since this happens in the first minute of the hour. We checked /var/log/cron log. There were a few jobs which ran close to the start of every hour. We turned those cron jobs off. Still no change. In the very first minute a spike starts. And lasts about 10-15 minutes.

Just as a side note. We are also seeing some dirtied inodes today. Also we have studied all recent deployments. We don't have any major changes.

Here is the server-status output just when the server starts hanging:

Current Time: Friday, 11-May-2012 21:02:42 UTC
Restart Time: Friday, 11-May-2012 19:09:28 UTC
Parent Server Generation: 0
Server uptime: 1 hour 53 minutes 13 seconds
Total accesses: 1128389 - Total Traffic: 8.9 GB
CPU Usage: u203.85 s36.51 cu.1 cs0 - 3.54% CPU load
166 requests/sec - 1.3 MB/second - 8.3 kB/request
500 requests currently being processed, 0 idle workers


Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

/var/log/messages continously receives text as follows. Notice how some words are corrupt. I wonder if there is a system mechanism which tries to fix something every hour.:

May 11 22:46:43 www kernel: <ald(2825): WRITE block urnald(2825): WRITE block 128852880 on sdb1kjournald(2825): WRITE block 12885<7>kjournald(2825): WRITE block 128852896 on sdb1
May 11 22:46:43 www kernel: >kjournald(2825): WRITE block 128853
May 11 22:46:43 www kernel:  WRITE block 128853104 on sdb5): WRITE block 128853112 on sdb1
May 11 22:46:43 www kernel: 2825): WRITE block 128853120 on sdb1
May 11 22:47:01 www kernel: ock 685806792 on sda1
May 11 22:47:05 www kernel: ock 129145376 on sdb1
May 11 22:47:10 www kernel: RITE block 789789472 on sda1
May 11 22:47:11 www kernel: ock 129189464 on sdb1
May 11 22:47:16 www kernel: <7ock 129360712 on sock 129360720 on sdb1
May 11 22:47:41 www kernel: <TE block 262819968 on sdb1

Where would you suggest I can track down the culprit which triggers this problem?

  • At the point this happens, what does netstat -an look like. – EightBitTony May 11 '12 at 23:09

Anything in the actual access log that points to external users going up? You might try running apachetop before the event and watching it as the event occurs.

| improve this answer | |
  • Nothing out of the ordinary within the access_log. I've tried apachetop per your advise. Nothing special. Since this happens at the top of the hour I suspect this is system operation trying to do some sort of clean up. – Haluk May 11 '12 at 22:03
  • Based on your new info above, I'd say someone turned on block_dump. try: sudo sysctl vm.block_dump=1 – roktechie May 12 '12 at 0:59

I have seen such spikes when faulty scripts made recursive requests. Can you check the access.log for requests the servers makes to itself (i.e. search for your server's IP address)?

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.