Apache Request Stuck

Question

Our company system is having slowness issue for the past week. After some debugging I was able to reproduce the problem by press and holding the F5 to let the page refresh numerous of time.

The page freeze and become unresponsive. I realize some request was stucked by looking at the server-status.

I did strace to the process and it shows the following message: "flock(16, LOCK_EX"

Here's the httpd.conf for prework.c andd keepalive was off:

<IfModule prefork.c>
StartServers       4
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000

</IfModule>

Seems like the request was deadlocked and stucked there forever, could this be the cause of the system slowness? Is it caused by some kind of mis-configuration?

ADD ON:

I also realize MySQL has process sleeping and doing nothing when the page is unresponsive. This will go away after 60 seconds which I believe is the time limit.

UPDATE http.conf:

<IfModule prefork.c>
StartServers       2
MinSpareServers    2
MaxSpareServers   10
ServerLimit      256
MaxClients       100
MaxRequestsPerChild  4000

</IfModule>

Answer 1

You have a webserver, and you have a database. You also have something bridging the two which you've not told us about. If the flock is owned by an Apache process then that's probably mod_Perl, mod_PHP or similar. And that's the first place you should be looking for the culprit. Certainly on PHP, the only thing which will lock files without an explicit call to flock() is the session handler. If you are using PHP and sessions, try a non locking handler otherwise check your code for calls to flock().

Note that in many scenarios it might not be a deadlock you are seeing but simply lots of slow, serialised requests.

Alternatively you might consider how to prevent the scenario you created from arising by limiting the number of connections and requests per client.

Answer 2

Your MaxClients is set too high. The formula to determine settings is:

(Total RAM – yMB)/xMB = MaxClients

where x is average Apache process size y is how much memory you want to keep available for other processes. With your numbers, and assuming even a small set aside of 500M for other processes, you would get:

(4000-500)/20=175

When your number of requests exceeds this, your server will start swapping to disk, response times will skyrocket into minutes, and finally your server will die a painful death. Note: a "request" is generated for every asset on a webpage (i.e., the .htm, each .css, each image file, etc. etc.), so just a few "page hits" (or maybe just one) can cause you to hit the limit!

Set it to 100 and see what happens. And while you are at, get your other settings back to the defaults, if not lower. For example:

StartServers 2
MinSpareServers    2
MaxSpareServers   10

are probably better.