-2

Our company system is having slowness issue for the past week. After some debugging I was able to reproduce the problem by press and holding the F5 to let the page refresh numerous of time.

The page freeze and become unresponsive. I realize some request was stucked by looking at the server-status.

server-status screenshot

I did strace to the process and it shows the following message: "flock(16, LOCK_EX"

Here's the httpd.conf for prework.c andd keepalive was off:

<IfModule prefork.c>
StartServers       4
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000

</IfModule>

Seems like the request was deadlocked and stucked there forever, could this be the cause of the system slowness? Is it caused by some kind of mis-configuration?

ADD ON:

I also realize MySQL has process sleeping and doing nothing when the page is unresponsive. This will go away after 60 seconds which I believe is the time limit.

Mysql Process List

UPDATE http.conf:

<IfModule prefork.c>
StartServers       2
MinSpareServers    2
MaxSpareServers   10
ServerLimit      256
MaxClients       100
MaxRequestsPerChild  4000

</IfModule>
  • What is your operating system? How much RAM is installed? What is your average Apache process size? – Colt May 9 '16 at 19:35
  • CentOS 6.6 Final 64bits, the RAM usage and CPU usage is low all the time. 4GB of RAM in total. Tot=472084(23) Avg=472084/23=20.0443MB Calculated by serverfault.com/questions/353828/… – June Leow May 9 '16 at 20:20
  • As we work through this, ADD to your original question, but don't change the question. You will make it impossible for other people to jump in and help and will make the question useless for others at a later time. – Colt May 9 '16 at 20:56
1

You have a webserver, and you have a database. You also have something bridging the two which you've not told us about. If the flock is owned by an Apache process then that's probably mod_Perl, mod_PHP or similar. And that's the first place you should be looking for the culprit. Certainly on PHP, the only thing which will lock files without an explicit call to flock() is the session handler. If you are using PHP and sessions, try a non locking handler otherwise check your code for calls to flock().

Note that in many scenarios it might not be a deadlock you are seeing but simply lots of slow, serialised requests.

Alternatively you might consider how to prevent the scenario you created from arising by limiting the number of connections and requests per client.

  • You're probably right, I called session_write_close() and seems like this lock go away. However my slowness problem is still there.. I created a static page that loads 5 images to test this out. The page still hangs like one out of 5 times. I will mark this as answer although the slowness problem is still here. – June Leow May 9 '16 at 23:52
1

Your MaxClients is set too high. The formula to determine settings is:

(Total RAM – yMB)/xMB = MaxClients

where x is average Apache process size y is how much memory you want to keep available for other processes. With your numbers, and assuming even a small set aside of 500M for other processes, you would get:

(4000-500)/20=175

When your number of requests exceeds this, your server will start swapping to disk, response times will skyrocket into minutes, and finally your server will die a painful death. Note: a "request" is generated for every asset on a webpage (i.e., the .htm, each .css, each image file, etc. etc.), so just a few "page hits" (or maybe just one) can cause you to hit the limit!

Set it to 100 and see what happens. And while you are at, get your other settings back to the defaults, if not lower. For example:

StartServers 2
MinSpareServers    2
MaxSpareServers   10

are probably better.

  • I set the MaxClients to 100 but seems like the slowness issue is getting more frequent than before. – June Leow May 9 '16 at 20:45
  • Adjust the other numbers, and then go lower on MaxClients. Your process size may higher than you are calculating, you may be using more than 512M on other processes, etc. While there might be other problems not addressed here, your MaxClients was without any question set too high (it is a mathematical calculation). Try 2, 2, 5, 30. – Colt May 9 '16 at 20:51
  • Updated httpd.conf, checkout the new config above. – June Leow May 9 '16 at 20:53
  • Hey, by the way: ADD to your original question, but don't change the question. You will make it impossible for other people to jump in and help and will make the question useless for others at a later time. – Colt May 9 '16 at 20:55
  • Updated the question. – June Leow May 9 '16 at 21:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.