Restarting nginx backends without losing requests

Question

I'm sure it's been asked before in different words but I run several Django sites via uwsgi (emporer mode) behind nginx. It's all a fairly standard configuration but I find that if I restart the central uwsgi process, nginx just bombs out 502s rather than waiting for the socket to become available.

I recognise that most of this is probably for a reason but people seeing 502 errors really stings me. It's certainly not something I want a client to see. So...

• Can I beg nginx to wait/retry backends? Or,
• Is there anything (other than the obvious) I can do to minimise commercial damage from uwsgi restarts?

Answer 1

One idea is to replace nginx's default 502 template with a page that automatically refreshes the client. You essentially just need to make a new file that has <meta http-equiv="refresh" content="5"> in the header. Give it some friendly text, explaining that the site is currently undergoing maintenance (or some equiv BS) and link to it from your nginx config:

error_page 502  /502.html;  
location = /502.html {  
    root  /var/www/502.html;  
}

You'll need that in all your sites (there may be a way of doing that globally) but the result is anybody who would see a gateway timeout will now see a page that doesn't look particularly odd and will, within five seconds, land them on the page they originally wanted.

That all assumes the backend will come back up. If there's a chance it will be off indefinitely, you might want to write something in JS that checks the URL itself and have a retry-counter. All fairly simple but it may appease clients who are getting annoyed with a site being down.

Answer 2

Just a thought, neither tested nor sure if possible:

What if you configure multiple upstream servers in nginx that all point to the same uWSGI instance. If nginx fails to communicate with uwsgi it will send the request to the next upstream (does proxy_next_upstream directive help here?), which is actually the same, but may be already up.