Nginx thinks php-fcgi process is dead, but script ran anyway

Question

Ok this is bit of a weird title, but the problem is weird, too.

We have a two server setup (www01 and www02) both running nginx and php as php-fcgi.

Nginx is configured to redirect the request to the other host if the php-fcgi process on the current host does not respond within 60 seconds.

What happened according to our logs: user is on www01 and posts a form to www01. The php-fcgi process does not respond and the request is forwarded by nginx to www02. php-fcgi again does not respond and the request is sent back to www01. Again php-fcgi does not respond and the request finally lands on www02 again. There it ends.

What is strange though: according to our database the form was indeed submitted AND the php application worked perfectly fine as the data submitted by the form was inserted 4 x (!) in the database each record created within a minute.

We couldnt find ANYTHING in the logs, except in the nginx error logs where nginx logged that it received upstream errors from www01/www02 as php-fcgi was not responding and forwarded the request to the other host. But no PHP error, no exception message, not even a warning.

How could this happen? Why did nginx think php-fcgi did not respond but the application ran anyway?

And how can we prevent this from happening again?

Answer 1

How much processing is your PHP script doing? If it takes e.g. 61 seconds to process that form data, nginx will of course see the 60 seconds elapse, assume something bad happened, and then send the request along to your other server.

How is PHP responding to the request? If it sends nothing at all, nginx might interpret that as "no response yet" and then again send the request along to the other server after waiting 60 seconds.

Try turning off the redirection in nginx and just see what happens when nginx is doing nothing more than very simply reverse-proxying your form submission to the single php-fcgi instance behind it. Simplifying your setup like this when mysterious things are going on is often a good way to zero in on what and where the unexpected results are coming from. If everything works fine in this simplified scenario, then I would suggest a close review of your redirection setup.

If you still can't figure it out, I suggest you post your nginx config so we can better help you out here; some more detail on the PHP script itself would help us help you, too.