4

I'm using NGINX 1.8.1 on CentOS 6 and 7. During our deploys, it is required that NGINX is stopped, the deploy executes, and NGINX starts again. Unfortunately, NGINX during a stop seems to just 500 all connections. As I'm using Ansible to update 2/10 hosts at a time, it's likely therefore that many of our clients will get 500'd throughout our deploy process.

Is there a way via NGINX settings to have the NGINX process drain all live connections (ie wait for them to complete) while rejecting all new connections during a stop?

I know that Amazon's Elastic Load Balancers do have this functionality, is there an equivalent for NGINX?

5
  • 1
    The upgrade process is similar, it may have some clues that you can use, but isn't an answer as such digitalocean.com/community/tutorials/…
    – Tim
    May 7, 2016 at 0:13
  • 1
    "[...] it is required that NGINX is stopped, the deploy executes, and NGINX starts again. [...]" Why is that?
    – gxx
    May 7, 2016 at 0:18
  • Bad downstream race condition that we cannot address. May 7, 2016 at 0:19
  • Besides: What about let the firewall block NEW connections but allow RELATED, ESTABLISHED?
    – gxx
    May 7, 2016 at 0:19
  • @gf_ that would make a lot of sense, can you provide an answer with some iptables magick? May 7, 2016 at 0:20

3 Answers 3

4

According to the comments, you could try to solve this using a firewall:

  • Task: Block incoming new connections, but keep related and established ones

  • Try: iptables -I INPUT -j DROP -p tcp --syn --destination-port 80 (or 443, whatever applies to your setup) to set the rule, thus blocking new traffic

  • Do your deployment...

  • Try: iptables -D INPUT -j DROP -p tcp --syn --destination-port 80 (or 443, whatever applies to your setup) to revert the rule, thus allowing new traffic

  • Note: I've just tested this shortly, it worked as expected. But: Please don't take this for granted, and test for yourself as well. Feedback welcome...

Edit: As @Guntram Blohm pointed out in the comments, better to use iptables -I instead of iptables -A. Changed this appropriately in the above code.

2
  • 1
    Warning: if there are already some iptables entries, they might handle the packet before it ever gets to your appended, i.e. last, rule. Better use -I instead of -A. May 7, 2016 at 7:15
  • @GuntramBlohm Thanks for your comment, good point! I've changed my answer accordingly.
    – gxx
    May 7, 2016 at 9:22
4

Send nginx the "QUIT" signal to perform a graceful shutdown, which stops listening for new connections but allows workers to continue to serve active connections.

It's worth distinguishing between graceful shutdown and connection draining; graceful shutdown affects how NGINX deals with incoming connections while the server is shutting down, but connection draining (which as Martijn Heemels mentioned is a feature of NGINX Plus) deals with how NGINX removes a backend server from service when acting as a load balancer. From your question, it sounds like you're more interested in the former.

1
  • As an aside, on CentOS 7, SystemD defines ExecStop=/bin/kill -s QUIT $MAINPID. It appears that on CentOS 6, the init script sends a TERM rather than QUIT, and this may be the source of our problems. May 8, 2016 at 6:05
3

As far as I can tell, connection draining is only available in the commercial version, Nginx Plus.

There's a description of the feature here: https://www.nginx.com/blog/nginx-plus-r5-released/

It's part of the ngx_http_upstream_conf_module module.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .