5

Alright, this is melting my brain. It might have something to do with the fact that I don't understand Upstart as well as I should. Sorry in advance for the long question.

I'm trying to use Upstart to manage a Rails app's Unicorn master process. Here is my current /etc/init/app.conf:

description "app"

start on runlevel [2]
stop on runlevel [016]

console owner

# expect daemon

script
  APP_ROOT=/home/deploy/app
  PATH=/home/deploy/.rbenv/shims:/home/deploy/.rbenv/bin:$PATH
  $APP_ROOT/bin/unicorn -c $APP_ROOT/config/unicorn.rb -E production # >> /tmp/upstart.log 2>&1
end script

# respawn

That works just fine - the Unicorns start up great. What's not great is that the PID detected is not of the Unicorn master, it's of an sh process. That in and of itself isn't so bad, either - if I wasn't using the automagical Unicorn zero-downtime deployment strategy. Because shortly after I send -USR2 to my Unicorn master, a new master spawns up, and the old one dies...and so does the sh process. So Upstart thinks my job has died, and I can no longer restart it with restart or stop it with stop if I want.

I've played around with the config file, trying to add -D to the Unicorn line (like this: $APP_ROOT/bin/unicorn -c $APP_ROOT/config/unicorn.rb -E production -D) to daemonize Unicorn, and I added the expect daemon line, but that didn't work either. I've tried expect fork as well. Various combinations of all of those things can cause start and stop to hang, and then Upstart gets really confused about the state of the job. Then I have to restart the machine to fix it.

I think Upstart is having problems detecting when/if Unicorn is forking because I'm using rbenv + the ruby-local-exec shebang in my $APP_ROOT/bin/unicorn script. Here it is:

#!/usr/bin/env ruby-local-exec
#
# This file was generated by Bundler.
#
# The application 'unicorn' is installed as part of a gem, and
# this file is here to facilitate running it.
#

require 'pathname'
ENV['BUNDLE_GEMFILE'] ||= File.expand_path("../../Gemfile",
  Pathname.new(__FILE__).realpath)

require 'rubygems'
require 'bundler/setup'

load Gem.bin_path('unicorn', 'unicorn')

Additionally, the ruby-local-exec script looks like this:

#!/usr/bin/env bash
#
# `ruby-local-exec` is a drop-in replacement for the standard Ruby
# shebang line:
#
#    #!/usr/bin/env ruby-local-exec
#
# Use it for scripts inside a project with an `.rbenv-version`
# file. When you run the scripts, they'll use the project-specified
# Ruby version, regardless of what directory they're run from. Useful
# for e.g. running project tasks in cron scripts without needing to
# `cd` into the project first.

set -e
export RBENV_DIR="${1%/*}"
exec ruby "$@"

So there's an exec in there that I'm worried about. It fires up a Ruby process, which fires up Unicorn, which may or may not daemonize itself, which all happens from an sh process in the first place...which makes me seriously doubt the ability of Upstart to track all of this nonsense.

Is what I'm trying to do even possible? From what I understand, the expect stanza in Upstart can only be told (via daemon or fork) to expect a maximum of two forks.

3 Answers 3

3

Indeed, a limitation of upstart is that it cannot track daemons that do what unicorn is doing.. that being fork/exec and exit their main process. Believe it or not, sshd does the same thing on SIGHUP, and if you look, /etc/init/ssh.conf makes sure sshd runs in the foreground. This is also one reason apache2 still uses an init.d script.

It sounds like gunicorn actually sort of daemonizes itself when receiving SIGUSR1 by forking and then exitting. This would be confusing for any of the process managers out there that try and keep a process alive.

I think you have two options. 1 is just to not use the SIGUSR1 and stop/start gunicorn when you need it.

The other option is to not use upstart's pid tracking, and just do this:

start on ..
stop on ..

pre-start exec gunicorn -D --pid-file=/run/gunicorn.pid
post-stop exec kill `cat /run/gunicorn.pid`

Not as sexy as pid tracking, but at least you won't have to write a whole init.d script.

(incidentally, this has nothing to do with the shebangs/execs. Both of those things work just like running a regular executable, so they wouldn't cause any extra forks).

1
  • 1
    I found this most useful also, please see my complete example, which also has a nice config for the deploy task. Jan 21, 2013 at 22:08
7

I picked little a different solution from SpamapS's.. I'm also running an app with preload_app = true, managed by Upstart.

When I was looking to solve this problem myself, I'd been using Upstart's "exec" to start my app ("exec bundle exec unicorn_rails blah blah"). Then I found your question, and it made me realize that instead of using Upstart's "exec" to specify my executable, I could use a script stanza, which would be run in its own process, the process that Upstart would watch.

So, my Upstart config file includes this:

respawn

script
  while true; do
    if [ ! -f /var/www/my_app/shared/pids/unicorn.pid ]; then
      # Run the unicorn master process (this won't return until it exits).
      bundle exec unicorn_rails -E production -c /etc/unicorn/my_app.rb >>/var/www/my_app/shared/log/unicorn.log
    else
      # Someone restarted the master; wait for the new master to exit.
      PID=`cat /var/www/my_app/shared/pids/unicorn.pid`
      while [ -d /proc/$PID ]; do
        sleep 2
      done
      # If we get here, the master has exited, either because someone restarted
      # it again (in which case there's already a new master running), or
      # it died for real (in which case we'll need to start a new process).
      # The sleep above is a tradeoff between polling load and mimizing the
      # restart delay when the master dies for real (which should hopefully be
      # rare).
    fi
  done
end script

The before_fork in my Unicorn config file is just as suggested in the example from the unicorn site, http://unicorn.bogomips.org/examples/unicorn.conf.rb:

before_fork do |server, worker|
  ActiveRecord::Base.connection.disconnect! if defined?(ActiveRecord::Base)

  old_pid = '/var/www/my_app/shared/pids/unicorn.pid.oldbin'
  if server.pid != old_pid
    begin
      sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
      Process.kill(sig, File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us
    end
  end
  sleep 0.5
end

So: at startup, the Upstart script doesn't find the pidfile, so it runs unicorn_rails, which keeps running.

Later, we redeploy our app, and a Capistrano task triggers app restart via:

kill -USR2 `cat /var/www/my_app/shared/pids/unicorn.pid`

This tells the old Unicorn master to start a new Unicorn master process, and as the new master starts workers, the Unicorn before_fork block sends TTOU signals to the old master to off the old workers (gracefully), then QUIT once there's only one worker left.

That QUIT causes the old master to exit (but only once there are new workers already handling the load), so the "bundle exec unicorn_rails" returns in the unicorn script. That script then loops around, sees the existing pidfile, and waits for the process to exit. It won't exit until the next deployment, but we'll loop around again if it does; we'd also loop around again anytime the master dies.

If the bash script itself dies, Upstart will restart it, because that's the process it's watching (as you see if you ever do status my_app -- Upstart reports the bash script's PID. You can still stop my_app, or restart my_app, which don't do any of the graceful stuff.

0
0

According to the upstart documentation, you can tell upstart to not send a TERM and KILL signals to a service by saying so in your pre-stop block. All you have to do is say start in your pre-stop and it will abort the signal sending.

Together with the trick above by Bryan who figured out that a script can also include an infinite loop and not be the actual unicorn process -- I created an example that works using this technique.

The example is fairly simple, it handles sending USR2 when running upstart stop unicorn. And it will also respawn a new unicorn if for some reason all unicorns died on their own.

Code and output of tests is here - https://gist.github.com/kesor/6255584

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .