We are having Apache processes get hung up after seeing a POST request, and wondering what the best path might be to troubleshooting this. When I look at the extended server status for the requests that hang, I see entries like:
53-31 28616 9/13/74232 W 0.71 7174 0 7.6 0.01 961.47 xxx.xxx.xxx.xxx www.mysite.com POST xxxxx HTTP/1.1
27-31 7629 1/34/107074 W 0.96 4480 0 0.0 0.39 1394.10 xxx.xxx.xxx.xxx www.mysite.com POST xxxxx HTTP/1.1
The process itself is always doing something, usually consuming significant CPU resources until we kill it.
Originally I assumed it was some bad code -- everything is running PHP scripts -- but after awhile we saw hung processes resulting from a variety of different requests, many of them simple, but always showing POST as the last type of request handled.
I don't expect that the community would instantly be able to solve this without seeing the actual code running; what I am really looking for is the best path to debug this as I am out of ideas. Some things that do come to mind:
Is there any kind of a tool similar to jstack that will allow us to dump and inspect a stack trace of a hung process, that would give useful information?
Is there a way to track useful input that might help troubleshoot this? I have tried setting the Apache logs to debug and having it output POST query parameters but the amount of data logged is overwhelming and I have a hard time knowing what to look for. Any best practice here?
The server is using prefork with MaxClients set to 150, and a KeepAlive of 5. Under normal load scenarios there are usually 30-50 processes up and running; at high load closer to 100. We have seen the hung process behavior occur under any load scenario. Is there any piece of the configuration that I might specifically look at?
The server is Apache 2 running under Debian Lenny on EC2.