6

I have a ruby on rails application which is running on passenger and nginx, with one main webserver and multiple application servers. I am trying to update my deployment process in order to minimize (or ideally, remove) any downtime caused by the deployment. The main roadblock right now is that passenger takes some time to restart (ie. reload the application), so in order to get around this, I want to stagger my restarts so that only one app server gets restart at a time.

In order to do this without losing any long running passenger processes, I am thinking I need to gracefully shutdown the app server's nginx instance, which will cause it to no longer accept new connections but continue to process the existing ones; as well, HAProxy will detect that the app server is down and route new requests to the other server.

However, assuming that there is a long-running process, I am not sure how to detect when the graceful shutdown has completed so that I can start it back up. Since the shutdown is caused by sending a signal (ie. kill -QUIT $( cat /var/run/nginx.pid )), and the kill command will return immediately, I cannot combine commands (ie. kill ... && touch restarted), as the touch command will execute immediately, even if nginx hasn't completed its shutdown.

Is there any good way to do this?

0

5 Answers 5

8

What about something like this?

while [ -n "$(pgrep nginx)" ]
do
  some-stuff  
done

So, the pgrep will look for any nginx processes, and the while loop will let it sit there until they are all gone. You can change some-stuff to do something useful, like sleep 1; /etc/init.d/nginx stop, so that it'll sleep for a second, then attempt to stop nginx using the init.d script. You can also put a counter in there somewhere so that you can bring out the heaving kill signals if it's taking too long.

2
  • Thanks! Based on this answer, I created a bash script to do the graceful restart for me (see my answer) Feb 7, 2012 at 22:12
  • If your nginx is proxy passing web sockets, make sure to turn off the process that is listening to the sockets before even gracefully quitting nginx, otherwise nginx never quite closes properly and this loop will get stuck forever.
    – Mahn
    Nov 8, 2015 at 20:16
4

Why not use the built in flag for this?

nginx -s quit

read the second block where it has this flag explained...

http://nginx.org/en/docs/control.html

3

Based on cjc's answer, here is the bash script I created, in case anyone else comes across the same problem:

#!/bin/bash
set -e

if [[ $(/usr/bin/id -u) -ne 0 ]]; then
  echo "This script must be run as root."
  exit
fi

if [ -n "$(pgrep nginx)" -a -f /var/run/nginx.pid ]; then
  PID=$( cat /var/run/nginx.pid )

  # Gracefully shutdown nginx
  kill -QUIT $PID

  # Wait for nginx to stop
  while [ -d "/proc/$PID" -a -f /var/run/nginx.pid ]; do
    sleep 1
  done

  # Restart nginx
  /usr/sbin/service nginx start
else
  echo "nginx is not running."
  exit 1
fi
1
  • I would do some sort of timer to get out of your while loop, in case nginx is disobedient about the signal and doesn't die in, say, a minute.
    – cjc
    Feb 7, 2012 at 22:37
1

You can check to see if /proc/$(cat /var/run/nginx.pid )/ exists. Or try and see if /var/run/nginx.pid gets removed upon shutdown.

0

ps -fC nginx

  root       7146     1  0 02:19 ?   00:00:00 nginx: master process /usr/sbin/nginx
  1. www     7147  7146  0 02:19 ?   00:00:00 nginx: worker process
  2. www     7148  7146  0 02:19 ?   00:00:00 nginx: worker process

so if you write a shutdown/restart script that loops until in this case 7146 has died, then restart nginx again

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .