I am running a CentOS 6.7 machine (16 GB vRAM, 8 vCPUs) with a simple web server setup (Apache/2.2.15, PHP/5.3.3, MySQL/5.1.66), hosting an online shop with a moderate number of page impressions (about 2,000 ~ 4,000 per day).
The server ran smooth over the last 3 years without any necessity to change its configuration. Now, since last week - somehow overnight - Apache/HTTP recurrently becomes inaccessible after a short period of time (about 30 minutes). I checked some parameters and saw that there are a lot of httpd processes running.
ps axf | grep httpd | wc shows something like this:
387 2344 18354
Whereas the load is not very exciting.
top looks like this:
A very small number of httpd processes is released again from time to time, but the total number keeps almost constantly increasing. If I run a
service httpd reload, the number of processes drops back to 0 and begins to increase again over the next couple of minutes to hours. After a while, Apache's log tells me:
[error] server reached MaxClients setting, consider raising the MaxClients setting
I did this and I also adjusted different further config params, but it did not help. No matter what value
ServerLimit are set to, Apache will not stop to spawn new httpd processes up to these limits. After this, the website is not accessible anymore.
There was no increase in page hits according to AWStats. Furthermore, there has been not a single change in the PHP application. Apart from the breakdown blocking all requests after 30 minutes, the website performs fast as usual. As a dirty and temporary workaround Cron keeps reloading Apache half-hourly.
Prefork/Worker settings have been as follows over the last 3 years:
<IfModule prefork.c> StartServers 4 MinSpareServers 5 MaxSpareServers 10 ServerLimit 128 MaxClients 128 MaxRequestsPerChild 600 </IfModule> <IfModule worker.c> StartServers 2 MaxClients 150 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 0 </IfModule>
I raised them up to:
<IfModule prefork.c> StartServers 8 MinSpareServers 5 MaxSpareServers 10 ServerLimit 640 MaxClients 640 MaxRequestsPerChild 1000 </IfModule> <IfModule worker.c> StartServers 4 MaxClients 300 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 0 </IfModule>
After this change, Apache/HTTP became unavailable after about 1 hr.
How can this happen all of a sudden and what options do I have to investigate further where this strange behaviour originates from?