Tell nginx to use different PHP process on random “502 Bad Gateway”

Question

On a webserver, that usually works quite well with nginx and php-fpm (currently version 7.0.22, but this is mostly independet from the version), there are times when a 502 Bad Gateway occurs. This sually means: One of the php-fpm processes has crashed, and nginx does not receive an answer from the chosen process.

My current workaround is to monitor not only the php-fpm process, but also the output of a PHP page. And if this does not work for 4 minutes (two failed retries with a surveillance interval of 2 minutes), then monit will kill all php-fpm processes and restart the php-fpm service. Works, but still causes a 5 minute downtime (at least for some users that connect to the broken process) or more, because monit may as well see the answer from a sane process a few times, before it observes the 502 Bad Gateway.

(1) The ideal solution was to bugfix whatever break the php-fpm process. Yet, the error occurs rarely, so I was unable to track it down to some specific reason. Possibly a memory leak in a PHP script ... I don't know.

(2) The second best option, I have in mind, would involve some cooperation from nginx. If the webserver process could react to a PHP failure, it could (a) kill the specifically broken process and (b) try another process instead of throwing the 502 Bad Gatway.

So far, I did not find an option to make nginx react to the failure. Who has an ideas how this could be achieved? Or is there an easier solution, I missed?

connect() to unix:/run/php/php7.0-fpm.sock failed (11: Resource temporarily unavailable)

Answer 1

You can't move the request from one process to another inside the same upstream, but you can move it from one upstream to another.

First of all, you need at least two upstreams (you will need two different php pools):

upstream yourproject {
    server unix:/run/php/php7.0-fpm_primary.sock;
    server unix:/run/php/php7.0-fpm_secondary.sock;
}

Then you can configure the failover between them:

location ~ \.php$ {
    # First, you need to capture fastcgi errors
    fastcgi_intercept_errors      on;

    # Specifies in which cases a request should be passed to the next server
    fastcgi_next_upstream         error timeout http_500 http_503;

    # Limits the time during which a request can be passed
    # to the next server
    fastcgi_next_upstream_timeout 10s;

    # Limits the number of possible tries for passing
    # a request to the next server
    fastcgi_next_upstream_tries   2;

    fastcgi_pass yourproject;
}

But this is not going to resolve your problem, because you would be doing the failover between the same server.

I suggest you:

1. Use TCP sockets instead of UNIX sockets, they are more reliable and stable for high concurrency:

   upstream yourproject {
       server 127.0.0.1:9000;
   }
   

   You have obviously point your php to this TCP socket in your php pool configuration (listen = 127.0.0.1:9000)

2. Increase process and fd limits /etc/limits.conf:

   *    soft    nofile    65536
   *    hard    nofile    65536
   *    hard    nproc     65536
   *    soft    nproc     65536
   

3. Increase net.core.somaxconn and net.core.netdev_max_backlog in /etc/sysctl.conf. Maybe you are reaching the limit.

4. If you are using pm = dynamic in your PHP configuration pool, check this parameters and adjust them according to the documentation and your system resources: pm.max_children, pm.start_servers, pm.min_spare_servers, pm.max_spare_servers. Maybe you are reaching the limit too.

5. Increase pm.max_requests in your PHP configuration to avoid the respawning process occur too often.

If this doesn't work, copy and paste your PHP and Nginx config files, to take them a deeper look.