Apache httpd processes overwhelming server

Question

Randomly Apache seems to freak out and eat up cpu usage and overwhelms the server. The amount of daily pageviews is only around 8000 so it can surely handle this traffic. It also seems to happen randomly. Has anyone heard of this happening before and possible solutions?

Some basic server stats: Running WHM/Cpanel latest stable, 4gig memory Xeon(R) CPU X3430 @ 2.40GHz.

Apache: 2.2.17 PHP: 5.2.17 w/ eAccelerator

Top:

top - 16:41:12 up 7:15, 1 user, load average: 2.77, 2.23, 1.91
Tasks: 79 total, 4 running, 75 sleeping, 0 stopped, 0 zombie
Cpu(s): 88.4% us, 2.5% sy, 0.0% ni, 9.1% id, 0.1% wa, 0.0% hi, 0.0% si
Mem: 4024216k total, 960384k used, 3063832k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 
31902 nobody 17 0 43072 16m 6804 S 65.9 0.4 0:07.93 httpd 
29802 nobody 16 0 45700 24m 13m R 32.3 0.6 0:19.77 httpd 
32159 nobody 15 0 41984 11m 3440 S 30.3 0.3 0:01.34 httpd 
28227 nobody 15 0 43492 17m 7456 S 29.6 0.4 0:27.59 httpd 
31848 nobody 16 0 46184 17m 6608 R 29.6 0.5 0:07.40 httpd 
31785 nobody 16 0 44872 16m 5964 S 26.0 0.4 0:01.05 httpd 
30310 nobody 15 0 43284 16m 7228 S 24.6 0.4 0:08.90 httpd 
32130 nobody 15 0 42064 12m 3776 S 21.6 0.3 0:02.18 httpd 
28597 nobody 16 0 43396 17m 7368 S 21.0 0.4 0:12.76 httpd 
32158 nobody 15 0 43180 14m 5216 S 21.0 0.4 0:01.00 httpd 
17861 nobody 16 0 43484 24m 14m S 17.3 0.6 1:15.55 httpd 
31764 mysql 15 0 631m 318m 5056 S 14.3 8.1 76:50.21 mysqld 
32157 nobody 15 0 42668 15m 6312 S 10.7 0.4 0:01.07 httpd 
29841 nobody 15 0 44512 18m 8128 S 7.7 0.5 0:15.14 httpd 
26296 nobody 15 0 44624 18m 7800 S 1.0 0.5 0:31.26 httpd 
31825 nobody 15 0 43268 16m 6468 S 0.7 0.4 0:07.50 httpd 
1 root 15 0 1716 608 520 S 0.0 0.0 0:00.49 init

Answer 1

A few things you can check:

• Is this server a VPS (the swap/buffers are 0k which typically happens on VPS servers)? If this is the case the slowdowns may be due, in part or in whole, to other sites on the server. If this is the case you can either switch VPS providers or move to a dedicated server.
• Check the output of /server-status since you have mod_status enabled. This will tell you what all your httpd clients are doing and may give you some idea of the cause of the issue (a slow loading page for example).
• Try benchmarking your site using ab (ApacheBench) to get your maximum capacity. 8000 hits/day is only a hit every 11 seconds but if the traffic is not evenly distributed or some of your pages take 10 seconds to render then you can easily get into trouble. You may just be hitting your site's maximum server capacity at times throughout the day.
• Try to get a more accurate count of your incoming traffic and don't assume you're only getting 8k hits/day. Google Analytics (or a similar service) or Apache log parsing would suffice and will let you know if one or more clients are hammering your site for some reason or if you just receive a sudden increase in traffic (for example, a blog post is featured on Reddit).
• Do the outages occur at specific times in the day (on the hour or at the same time each day)? If so it may be a cron script that is causing the issue.