Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a ruby on rails application which is running on passenger and nginx, with one main webserver and multiple application servers. I am trying to update my deployment process in order to minimize (or ideally, remove) any downtime caused by the deployment. The main roadblock right now is that passenger takes some time to restart (ie. reload the application), so in order to get around this, I want to stagger my restarts so that only one app server gets restart at a time.

In order to do this without losing any long running passenger processes, I am thinking I need to gracefully shutdown the app server's nginx instance, which will cause it to no longer accept new connections but continue to process the existing ones; as well, HAProxy will detect that the app server is down and route new requests to the other server.

However, assuming that there is a long-running process, I am not sure how to detect when the graceful shutdown has completed so that I can start it back up. Since the shutdown is caused by sending a signal (ie. kill -QUIT $( cat /var/run/nginx.pid )), and the kill command will return immediately, I cannot combine commands (ie. kill ... && touch restarted), as the touch command will execute immediately, even if nginx hasn't completed its shutdown.

Is there any good way to do this?

share|improve this question

5 Answers 5

up vote 5 down vote accepted

What about something like this?

while [ -n "$(pgrep nginx)" ]
do
  some-stuff  
done

So, the pgrep will look for any nginx processes, and the while loop will let it sit there until they are all gone. You can change some-stuff to do something useful, like sleep 1; /etc/init.d/nginx stop, so that it'll sleep for a second, then attempt to stop nginx using the init.d script. You can also put a counter in there somewhere so that you can bring out the heaving kill signals if it's taking too long.

share|improve this answer
    
Thanks! Based on this answer, I created a bash script to do the graceful restart for me (see my answer) –  Daniel Vandersluis Feb 7 '12 at 22:12

You can check to see if /proc/$(cat /var/run/nginx.pid )/ exists. Or try and see if /var/run/nginx.pid gets removed upon shutdown.

share|improve this answer

Based on cjc's answer, here is the bash script I created, in case anyone else comes across the same problem:

#!/bin/bash
set -e

if [[ $(/usr/bin/id -u) -ne 0 ]]; then
  echo "This script must be run as root."
  exit
fi

if [ -n "$(pgrep nginx)" -a -f /var/run/nginx.pid ]; then
  PID=$( cat /var/run/nginx.pid )

  # Gracefully shutdown nginx
  kill -QUIT $PID

  # Wait for nginx to stop
  while [ -d "/proc/$PID" -a -f /var/run/nginx.pid ]; do
    sleep 1
  done

  # Restart nginx
  /usr/sbin/service nginx start
else
  echo "nginx is not running."
  exit 1
fi
share|improve this answer
    
I would do some sort of timer to get out of your while loop, in case nginx is disobedient about the signal and doesn't die in, say, a minute. –  cjc Feb 7 '12 at 22:37

ps -fC nginx

  root       7146     1  0 02:19 ?   00:00:00 nginx: master process /usr/sbin/nginx
  1. www     7147  7146  0 02:19 ?   00:00:00 nginx: worker process
  2. www     7148  7146  0 02:19 ?   00:00:00 nginx: worker process

so if you write a shutdown/restart script that loops until in this case 7146 has died, then restart nginx again

share|improve this answer

Why not use the built in flag for this?

nginx -s quit

read the second block where it has this flag explained...

http://nginx.org/en/docs/control.html

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.