4

I thought MaxClients is just a limit and when the server hits the limit, it shows blank pages for some people preventing the overload. Also I thought it's just a limit, so the spare connections aren't used unless they are needed and they are freed after that. But my server machine seems to deny all this.

Several months ago there where repeatedly "consider rising MaxClient" line in Apache's log of my server machine. Before I realized what's happening, server pile up the CPU load and then the machine crashed completely. That happens like 3 times in about a week before I rise up the MaxClient value from 500 to 900 (MPM prefork).

[Last crash*] Long after, 2 weeks ago, there were a connection peak caused by real visits (we published some "exlusive" info). It probably hits the 900 limit (it showed up once at log), server start to lag and in matter of minutes it crashed again. That end with 2 hours blackout (filesystem has got corrupted).

So now I have MaxClient = 2000 but the problem I have had, appeared days before the [Last crash*]. So I actually don't know if it is relevant.

These are my symptoms:

  • random CPU load increase (up to like 12, usual load under 5) until restarting Apache, after restart load drops down rapidly
  • after restart it tooks from hours to days before it shows up again
  • this load increase seems to add like "5" to existing load, meaning, in the morning it's approx. 2+5 and if I won't restart Apache it goes higher with real visits (12-14 at peak)
  • when it hits the peak, it won't go down, it stay there until restart no matter day time or actual visits
  • it seems the load issues start at random times, peaks, night, morning, Monday, Saturday doesn't matter
  • this load doesn't seem to affect performance at all (but I don't like it, if nothing else, it is shortening the CPU lifetime)
  • the load is caused by Apache for sure (top, htop checks that)

This is my MPM prefork setting (24GB RAM, Apache + MySQL + data):

StartServers          5
MinSpareServers       5
MaxSpareServers      10
ServerLimit   2000
MaxClients         2000
MaxRequestsPerChild   0

Also I have KeepAlive off. Could that could play some role at this behaviour?

KeepAlive Off
MaxKeepAliveRequests 0
KeepAliveTimeout 2

From server status: 40 requests currently being processed, 0 (or up to like 12) idle workers now. It's saturday night tho. Now I'm getting really similar numbers but it's sunny Sunday noon... damn it xD

First I thought it's faulty php script but I'm not sure about anything now. No one seems to change any script when the problem showed up first time. And the weird crashing behaviour when hitting the limit tells me, there could be wrong something entirely else (especially when the rising value helps, it doesn't look like memory lack).

I'm sorry for the exhausting description. I don't usually ask before search but I seriously ran out of ideas today. I can't keep restarting all the time. These 2 seconds doesn't matter but there is also the chance some major script couldn't finish it's job properly or editor lost his work when hitting the save button at the wrong moment.

So, to keep it simple, I would like your opinion on my Apache's setting as well if it's even possible that this could have some relevance with my load issues.

1
  • That load shortens CPU lifetime is a myth. Mar 18, 2012 at 19:26

3 Answers 3

4

Even with 24 GB RAM your MaxClients is probably way too high. This is what I use for an approximation of the average httpd (substitute apache2 if on Debian distro) process size:

ps -ylC httpd --sort:rss | awk '{sum+=$8; ++n} END {print "Tot="sum"("n")";print "Avg="sum"/"n"="sum/n/1024"MB"}'

Once you have that figure take say 90% of the system's RAM (assuming this server is only serving web traffic and doesn't have MySQL and other services on it) and divide by the average httpd process size, e.g.:

22,118 MB / 50 = 442

So you'd set MaxClients (and ServerLimit) to around 440.

Since you have MySQL on this server as well you would need to be more conservative, maybe take 50% of the system's RAM and divide it by the average httpd process size resulting in a MaxClients of 245.

5
  • It is Debian, I'm getting approx. 56MB. I knew it is probably too high but do I have a choice? More than 24GB RAM looks like wasting and this machine can't handle it anyway. Setting MaxClients to 245 will result into server crashes mentioned above. How could I prevent them? Weird thing is, load rise up even though there is almost zero traffic. So there is a little chance the MaxClient doesn't have to do anything with my issues but I would like to set these values right anyway. Buying dedicated machine with at least 48GB RAM doesn't sound right. There is no more than 20k visits/day.
    – Saix
    Mar 18, 2012 at 12:53
  • You said the load is caused by Apache for sure. How many CPUs/cores do you have on this machine? What is your web application? Native PHP? Drupal? Wordpress? Are you doing any caching? APC?
    – HTTP500
    Mar 18, 2012 at 13:48
  • 1
    When this happens there are like 5 apache processes with almost 100% CPU usage until restart. I have 8 CPU cores, APC script caching correctly enabled. Whole application is our own, no third-party cms like WP or Drupal.
    – Saix
    Mar 18, 2012 at 13:56
  • Well you may have to profile your code (e.g. xdebug) to see where it is spending its time.
    – HTTP500
    Mar 18, 2012 at 14:28
  • @HTTP500 This should be the accepted answer--good stuff. +1
    – Elliot B.
    Sep 24, 2012 at 4:24
1

Try this:

Timeout     300
KeepAlive   On
MaxKeepAliveRequests    28800
KeepAliveTimeout    260
ListenBackLog   default
StartServers    4
ServerLimit     16
ThreadLimit     2048
MaxClients  2048
MinSpareThreads     32
MaxSpareThreads     64
ThreadsPerChild     128
MaxRequestsPerChild     0
0

As a workaround, you could try setting MaxRequestsPerChild to a non-zero value.

3
  • What value would you suggest? There are too many different views on this topic. Debian default was 0, if there is no value at all, it should be 10000. But this usually cause load problems, so people lower that to 100 or 200 but they probably not have the same traffic and machine, so I am kinda confused. Should I set these values like MaxRequestsPerChild*MaxClients=2000? Consider I actually need those 2k connections.
    – Saix
    Mar 18, 2012 at 13:14
  • @Saix: MaxRequestsPerChild has nothing to do with the number of concurrent requests the server is capable of handling. It just means that after a certain amount of requests, the child exists, thus it's useful as a workaround for memory leaks, application hangs and other bugs. You need to determine what is a suitable value for your environment.
    – janneb
    Mar 18, 2012 at 15:45
  • It tooks a long time but it's back again and id doesn't seem to be a memory leak ( link ) It has to be script fault. Debugging wouldn't be easy, near impossible. Damn, I would really appreciate if there would be the way to see what script is wrapped by these Apache's processes.
    – Saix
    Mar 24, 2012 at 12:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .