Ok this is bit of a weird title, but the problem is weird, too.
We have a two server setup (www01 and www02) both running nginx and php as php-fcgi.
Nginx is configured to redirect the request to the other host if the php-fcgi process on the current host does not respond within 60 seconds.
What happened according to our logs: user is on www01 and posts a form to www01. The php-fcgi process does not respond and the request is forwarded by nginx to www02. php-fcgi again does not respond and the request is sent back to www01. Again php-fcgi does not respond and the request finally lands on www02 again. There it ends.
What is strange though: according to our database the form was indeed submitted AND the php application worked perfectly fine as the data submitted by the form was inserted 4 x (!) in the database each record created within a minute.
We couldnt find ANYTHING in the logs, except in the nginx error logs where nginx logged that it received upstream errors from www01/www02 as php-fcgi was not responding and forwarded the request to the other host. But no PHP error, no exception message, not even a warning.
How could this happen? Why did nginx think php-fcgi did not respond but the application ran anyway?
And how can we prevent this from happening again?