Apache and high load avg

Question

I have an odd problem I am not sure how to analyze further - there is a pretty lighweight website running on a pretty good box(Xeon, 4Gb ram, SATA). Judging by access logs, it get's about 120 requests per second.

Load avg. on a box is jumping between 5 and 20 pretty much randomly, but rarely drops below 10. CPU utilization is at about 70% avg and is quite consistent. There is a lot of apache processes and they all seem to be locking at some point on something which causes load. avg to jump above 15, I can't quite figure out how to find out what they locking on.

What can I use to see what's going on inside apache process itself and where it blocks?

Answer 1

strace one of the running processes can help

netstat -n |grep :80|cut -c 45-|cut -f 1 -d ':'|sort|uniq -c|sort -nr|more

(might need to adjust the 45- depending on the version of linux)

or perhaps:

tail -n 10000 access.log|cut -f 1 -d ' '|sort|uniq -c|sort -nr|more

do you have any busy IPs?

If CPU is 70%, that means it is computational load rather than a file lock. I would suspect a php script that is doing a lot of calculations in order to display a page. Perhaps a recursive include if you see the server's IP show up high on either of those lists. Or, if you have include('http://localserver.com/file.php'); sometimes that can drive the load higher because php must include that file through a socket.

Are you approaching MaxClients? anything in the apache error log?

Answer 2

Do any of the apache processes seem to be taking a lot of the cpu when viewed in top? If so, try to strace one of them to see what it is doing. It'll provide quite a bit of noise, but, you might be able to see a pattern. You can use script strace -p processid to watch it, then, control-d at the end and have a file named typescript which contains all of the information as well.

I suspect you have a php application that is probably burning a lot of CPU either doing massive calculations or even something as simple as traversing a directory for a file that has tens of thousands of files. It is also possible that you might have a mod_rewrite that is doing a number of redirections. Based on the fact that the cpu load is rather constant, it could be something locking a file, but, it seems more like you've got a recursive fetch or something triggered based on someone hitting a page.

Answer 3

Based on op's comment, you should tune down the keep alive timeouts, typically they are quite high on default configurations.

Also, if you have that many connections per second and keep alive timeouts are high, you are running into a risk of running out of file descriptors which could lead to io waits.