I'm using NGINX 1.8.1 on CentOS 6 and 7. During our deploys, it is required that NGINX is stopped, the deploy executes, and NGINX starts again. Unfortunately, NGINX during a stop seems to just 500 all connections. As I'm using Ansible to update 2/10 hosts at a time, it's likely therefore that many of our clients will get 500'd throughout our deploy process.

Is there a way via NGINX settings to have the NGINX process drain all live connections (ie wait for them to complete) while rejecting all new connections during a stop?

I know that Amazon's Elastic Load Balancers do have this functionality, is there an equivalent for NGINX?

  • 1
    The upgrade process is similar, it may have some clues that you can use, but isn't an answer as such digitalocean.com/community/tutorials/… – Tim May 7 '16 at 0:13
  • 1
    "[...] it is required that NGINX is stopped, the deploy executes, and NGINX starts again. [...]" Why is that? – gf_ May 7 '16 at 0:18
  • Bad downstream race condition that we cannot address. – Naftuli Kay May 7 '16 at 0:19
  • Besides: What about let the firewall block NEW connections but allow RELATED, ESTABLISHED? – gf_ May 7 '16 at 0:19
  • @gf_ that would make a lot of sense, can you provide an answer with some iptables magick? – Naftuli Kay May 7 '16 at 0:20

According to the comments, you could try to solve this using a firewall:

  • Task: Block incoming new connections, but keep related and established ones

  • Try: iptables -I INPUT -j DROP -p tcp --syn --destination-port 80 (or 443, whatever applies to your setup) to set the rule, thus blocking new traffic

  • Do your deployment...

  • Try: iptables -D INPUT -j DROP -p tcp --syn --destination-port 80 (or 443, whatever applies to your setup) to revert the rule, thus allowing new traffic

  • Note: I've just tested this shortly, it worked as expected. But: Please don't take this for granted, and test for yourself as well. Feedback welcome...

Edit: As @Guntram Blohm pointed out in the comments, better to use iptables -I instead of iptables -A. Changed this appropriately in the above code.

  • Warning: if there are already some iptables entries, they might handle the packet before it ever gets to your appended, i.e. last, rule. Better use -I instead of -A. – Guntram Blohm May 7 '16 at 7:15
  • @GuntramBlohm Thanks for your comment, good point! I've changed my answer accordingly. – gf_ May 7 '16 at 9:22

Send nginx the "QUIT" signal to perform a graceful shutdown, which stops listening for new connections but allows workers to continue to serve active connections.

It's worth distinguishing between graceful shutdown and connection draining; graceful shutdown affects how NGINX deals with incoming connections while the server is shutting down, but connection draining (which as Martijn Heemels mentioned is a feature of NGINX Plus) deals with how NGINX removes a backend server from service when acting as a load balancer. From your question, it sounds like you're more interested in the former.

  • As an aside, on CentOS 7, SystemD defines ExecStop=/bin/kill -s QUIT $MAINPID. It appears that on CentOS 6, the init script sends a TERM rather than QUIT, and this may be the source of our problems. – Naftuli Kay May 8 '16 at 6:05

As far as I can tell, connection draining is only available in the commercial version, Nginx Plus.

There's a description of the feature here: https://www.nginx.com/blog/nginx-plus-r5-released/

It's part of the ngx_http_upstream_conf_module module.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.