apache with php fastcgi keeps going down

Question

I have an apache2 server configured with MPM worker and php fast cgi. Lately the apache logs have been telling me that MaxClients is being reached frequently, even though it's already pretty high.

My server is now constantly going down, and I see a bunch of lines like this in the log:

[Sun Mar 06 04:25:40 2011] [error] [client 50.16.83.115] FastCGI: comm with (dynamic) server "/var/local/fcgi/php-cgi-wrapper.fcgi" aborted: (first read) idle timeout (20 sec)
[Sun Mar 06 04:25:40 2011] [error] [client 50.16.83.115] FastCGI: incomplete headers (0 bytes) received from server "/var/local/fcgi/php-cgi-wrapper.fcgi"

I can see that my php-cgi processes are pretty large (about 70mb on average).

Here's my apache configuration for MPM worker: KeepAlive ON KeepAliveTimeout 2

<IfModule mpm_worker_module>
    StartServers         5
    MinSpareThreads      10
    MaxSpareThreads      10
    ThreadLimit          64
    ThreadsPerChild      10
    MaxClients           20
    MaxRequestsPerChild  2000
</IfModule>

Heres my fastcgi apache configuration:

<IfModule mod_fastcgi.c>
  # One shared PHP-managed fastcgi for all sites
  Alias /fcgi /var/local/fcgi
  # IMPORTANT: without this we get more than one instance
  # of our wrapper, which itself spawns 20 PHP processes, so
  # that would be Bad (tm)
  FastCgiConfig -idle-timeout 20 -maxClassProcesses 1
  <Directory /var/local/fcgi>
    # Use the + so we don't clobber other options that
    # may be needed. You might want FollowSymLinks here
    Options +ExecCGI
  </Directory>
  AddType application/x-httpd-php5 .php
  AddHandler fastcgi-script .fcgi
  Action application/x-httpd-php5 /fcgi/php-cgi-wrapper.fcgi
</IfModule>

Here's my fastcgi wrapper:

#!/bin/sh
PHPRC="/etc/php5/apache2"
export PHPRC
PHP_FCGI_CHILDREN=8
export PHP_FCGI_CHILDREN

exec /usr/bin/php-cgi

Any help would be very very much appreciated!

Answer 1

I don't think your server configuration is related to your problem, but let's tackle this first : mod_fastcgi will take care of spawning PHP processes and only feed one request at once for each process (AFAIK, I'm a mod_fcgid user). Thus it is useless to ask PHP to fork other processes, you'd rather have PHP_FCGI_CHILDREN=0 to turn this feature off.

Lately the apache logs have been telling me that MaxClients is being reached frequently, even though it's already pretty high.

Your MaxClients is 20, on my MPM-worker sites it's in the 5000-10,000 ballpark ! This is very low.

Keep in mind that MaxClients only regulates the Apache process pool, and Apache scales very well up to serving thousands of concurrent connections.

You can regulate separately your FastCGI process pool with mod_fastcgi options. I can only help with solid experience on mod_fcgid, sorry. This process pool has most of the time a size which is between 1x and 4x the number of CPU cores.

[Sun Mar 06 04:25:40 2011] [error] [client 50.16.83.115] FastCGI: comm with (dynamic) server "/var/local/fcgi/php-cgi-wrapper.fcgi" aborted: (first read) idle timeout (20 sec)

Your application could not send a single byte in 20 seconds, that's pretty slow. You will have to find the bottleneck. As a rule of thumb, if the load average is high, there's a good change it's CPU or I/O bound (then proceed to check CPU usage); otherwise if load is low, check waits on remote ressources (database server, 3rd part APIs, and so on).

Answer 2

Your scripts are taking long than 20 seconds. Either increase the idle timeout value or investigate why they are taking all that time (eg. waiting for DB connections).

Answer 3

As far as I can tell, there was a problem with one of the extensions I had installed.

https://github.com/mreiferson/php-wkhtmltox/issues/11

Answer 4

You can solve this by increasing number of PHP_FCGI_CHILDREN until it'll eat up all of you memory or debug the problem, why the scripts take too long to execute. Common problems are: slow DB, slow external request (with curl or fopen).