Diagnosing apache process using 100% CPU

Question

I have a webserver setup to provide vhost-based web hosting for a bunch of domains, and has been working fine for a while. The server is running Ubuntu with Apache. PHP is running as FastCGI and not an Apache module.

Since Sunday morning the apache2 process will occassionally hit 100% CPU utilisation and stick there, no longer serving any requests until it is reset. A simple restart of apache resolves the problem.

All the sites are set up to use FastCGI for the PHP interpreter, so the most obivous source of the problem that I suspected (bad PHP scripts) should've resulted in php5-cgi capping the CPU, not apache2, right?

Is there anything that can be done to interrogate the apache2 process in some way, to see what it is doing at this point?

Simple web-log investigations don't show any suspicious activity or ramping request-flooding.

Bit stumped - currently just running a monitoring script that checks for a rampant apache2 process and restarts it if found.

---

Update:

The process pegged 100% again this morning, and a quick strace -f -p [pid] resulted in the following output. These 6 lines repeat as fast as the screen will scroll.

wait4(3437, 0x7fffaf867184, WNOHANG|WSTOPPED, NULL) = 0
wait4(5497, 0x7fffaf867184, WNOHANG|WSTOPPED, NULL) = 0
wait4(5504, 0x7fffaf867184, WNOHANG|WSTOPPED, NULL) = 0
wait4(3436, 0x7fffaf867184, WNOHANG|WSTOPPED, NULL) = 0
wait4(4035, 0x7fffaf867184, WNOHANG|WSTOPPED, NULL) = 0
wait4(3437, 0x7fffaf867184, WNOHANG|WSTOPPED, NULL) = 0

Answer 1

You can try attaching to the rogue process using strace:

strace -f -p [PID]

This will show you which syscalls the process is making, if any. Do note that the process may not be wasting its CPU time in syscalls but performing actual in-process work instead, but if the process entered a loop that involves a syscall someplace this will give you valuable information.