How do I automatically restart a process on server reboot/process exit?

Question

On my Ubuntu server I use God to monitor some Ruby processes/scripts, including a Queue system.

God automatically checks when one of my processes exists and restarts it. It also keeps track of memory usage and acts properly when specific conditions are triggered. Today I rebooted the server and after a while I noticed my queue wan't working. After a quick check, I discovered God wasn't restarted after server reboot (I supposed it should).

What is the best way to ensure God (or a generic process) is automatically restarted on server reboot or when the process is killed for whatever reason?

The specific command I use to start God is

god -c config

Answer 1

Several options here. Just skip to the last paragraph if you want one recommendation instead of reading about the others too!

Just restarting on reboot is easy: add it to /etc/rc.local which Debian based distros (and many others) run after everything in the other startup scripts by default. If god doesn't drop into the background itself you might need to att & to the ned of the command like so:

god -c config &

The above will not restart the process when it is killed though. You could write a script the infinite loops but that is messy and will cause problems when you really want it killed.

You could also create your own init script, modelled on those in /etc/init.d (and linked into /etc/rc.*' which will give you a more complete solution as you will have facility to stop/start the service when you need to, though this will not automatically start the service for you without some extra scripting on your part.

You could add it to your /etc/inittab, then init will manage the process for you, starting it on boot and restarting it if it dies. It will even stop restarting it if it dies to often in a short space of time. This is still not perfect though.

I think the generally recommended approach for what you have described is daemontools (http://en.wikipedia.org/wiki/Daemontools) which also supports restarting the service if it dies. This is a little more work here than using an entry in /etc/initab but you will end up with a much cleaner end product that should do everything you asked in your question. daemontools should be found in the repositories of just about any Linux distribution.

Answer 2

If you to ensure the process is always running then inittab is the sensible way to do it as it will restart the process if it dies. However I would not do it this way as if it is failing I would rather discover this sooner rather than later as my code would improve and I would be less likely to make the machine unstable.

If you have root access then create a init script in /etc/init.d and then use update-rc.d to create the correct symbolic links in the run level directories. An example script would look like

 #!/bin/bash
 ### BEGIN INIT INFO
 # Provides:          god
 # Required-Start:    $syslog
 # Required-Stop:     $syslog
 # Default-Start:     2 3 4 5
 # Default-Stop:      0 1 6
 # Short-Description: Runs the ruby god meta server.
 # Description:       Runs the ruby god meta server.
 ### END INIT INFO

 start_daemon () {

        nohup start-stop-daemon --start --quiet -m --pidfile /var/run/god.pid \
        --exec  god -- -c config 
        RETVAL=$?
 }

 stop_daemon () {
    start-stop-daemon --stop --quiet --oknodo -m --pidfile /var/run/god.pid 
    RETVAL=$?
 }


 case "$1" in 

 start)
 start_daemon
 ;;
 stop)
 stop_daemon
 ;;
 restart | reload | force-reload )
 stop_daemon
 start_daemon
 ;;
 esac

If you are just a normal user then you could use cron to run processes at boot time by specifing the start time as @reboot

@reboot god -c config

Answer 3

You need to write an init.d script for it to start and stop god. You can use any script under /etc/init.d as an example. Call your new script /etc/init.d/god. You can then run

# update-rc.d god defaults

and it should make sure it starts and stops at server boot and shutdown. You may find that there is already an init.d script for god available on the internet. I'd have a quick search before you sit down to write one.

Answer 4

To ensure god restarts it on boot, you'll want to add it to init scripts.. To do this, write a script that will accept at least start as a parameter (to follow standards, you'd have to do somewhat more than that) and stick it in /etc/init.d/ . Then add a symlink called S99 to /etc/rc[whatever run levels you want the script to start at ].

Ideally, to better follow standards, copy /etc/init.d/skeleton to somewhere else in /etc/init.d/ and edit it to fit.