Nginx config reload without downtime

Question

I use nginx as a reverse proxy. Whenever I update the config for it using

sudo "cp -r #{nginx_config_path}* /etc/nginx/sites-enabled/"
sudo "kill -s HUP `cat /var/run/nginx.pid`"

I face a brief downtime. How can I avoid that?

Answer 1

Run service nginx reload or /etc/init.d/nginx reload

It will do a hot reload of the configuration without downtime. If you have pending requests, then there will be lingering nginx processes that will handle those connections before it dies, so it's an extremely graceful way to reload configs.

Sometimes you may want to prepend with sudo

Answer 2

Run /usr/sbin/nginx -s reload

See http://wiki.nginx.org/CommandLine for more command line options.

Answer 3

Nginx and Signals

The kill approach you used (kill -s HUP $(cat /var/run/nginx.pid) is correct. Init scripts for RH or Debian distributions are in the end also implemented using kill command. You can check Init example from nginx website or contents of Ubuntu Nginx package.

There are multiple signals, that nginx can listen to (mentioned in wiki):

• TERM, INT - Quick shutdown.
• QUIT - Graceful shutdown.
• KILL - Halts a stubborn process.
• HUP - Configuration reload. Start the new worker processes with a new configuration. Gracefully shutdown the old worker processes.
• USR1 - Reopen the log files.
• USR2 - Upgrade Executable on the fly.
• WINCH - Gracefully shutdown the worker processes.

Nginx Reload

Nginx reload (HUP signal) is more specifically implemented as several steps [1,2]:

• The master process checks the syntax validity.
• Applies new configuration, that is, to open log files and new listen sockets.
• If this fails, it rolls back changes and continues to work with old configuration.
• If this succeeds, it starts new worker processes, and sends messages to old worker processes requesting them to shut down gracefully.
• Old worker processes close listen sockets and continue to service old clients.
• After all clients are serviced, old worker processes are shut down.

Only one issue I can think of why you had downtime (based on the reload process) is that you were using only one worker process (worker_processes directive), which by design was serving old clients, but had closed listen socket, therefore you couldn't open new connection.

I can also recommend you to always use /usr/sbin/nginx -t to validate configuration files before applying new config.

Nginx Reload in Depth

Reconfigure signal is handled in file ngx_process_cycle.c and we can see it starts new worker processes in function ngx_start_worker_processes(...) and at the end it stops old worker processes in function ngx_signal_worker_processes(...), which iterates over them with NGX_SHUTDOWN_SIGNAL signal.

Resources:

• [1] https://nginx.org/en/docs/control.html
• [2] https://www.nginx.com/resources/wiki/start/topics/tutorials/commandline/#stopping-or-restarting-nginx

Answer 4

No, you are incorrect, you aren't supposed to be facing any downtime with the procedure you describe. (Nginx can do not only configuration reload on the fly without any downtime, but even the upgrade of the executable on the fly, still without any downtime.)

As per http://nginx.org/docs/control.html#reconfiguration, sending the HUP signal to nginx makes sure that it performs a graceful restart, and, if the configuration files are incorrect, the whole procedure is abandoned, and you're left with the nginx as before sending the HUP signal. At no point should any downtime be possible.

In order for nginx to re-read the configuration file, a HUP signal should be sent to the master process. The master process first checks the syntax validity, then tries to apply new configuration, that is, to open log files and new listen sockets. If this fails, it rolls back changes and continues to work with old configuration.

Answer 5

For completeness, the systemd way of doing it:

systemctl reload nginx

Answer 6

Usually, reloading configuration file of a service should not affect the running service. However, this depends on how the SIGHUP signal is processed.

If a specific service is experiencing a downtime during reload, this can be circumvented by running the same service on multiple servers preferably using a load balancer. In this case, you can take out one server at a time and reload/restart it. Then, it can be re-added after confirming it is OK.