Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I want Upstart to do two things:

  1. stop trying to respawn a failed process so fast
  2. never give up trying to respawn

In an ideal world, upstart would try to restart a dead process after 1s, then double that delay on each attempt, until it reached an hour.

Is something like this possible?

share|improve this question

4 Answers 4

up vote 13 down vote accepted
+50

The Upstart Cookbook recommends a post-stop delay (http://upstart.ubuntu.com/cookbook/#delay-respawn-of-a-job). Use the respawn stanza without arguments and it will continue trying forever:

respawn
post-stop exec sleep 5

(I got this from http://askubuntu.com/questions/52309/respawning-too-fast-stopped-how-to-make-upstart-disable-for-5-mins.)

To add the exponential delay part, I'd try working with an environment variable in the post-stop script, I think something like:

env SLEEP_TIME=1
post-stop script
    sleep $SLEEP_TIME
    NEW_SLEEP_TIME=`expr 2 * $SLEEP_TIME`
    if [ $NEW_SLEEP_TIME -ge 60 ]; then
        NEW_SLEEP_TIME=60
    fi
    initctl set-env SLEEP_TIME=$NEW_SLEEP_TIME
end script

** EDIT **

To apply the delay only when respawning, avoiding the delay on a real stop, use the following, which checks whether the current goal is "stop" or not:

env SLEEP_TIME=1
post-stop script
    goal=`initctl status $UPSTART_JOB | awk '{print $2}' | cut -d '/' -f 1`
    if [ $goal != "stop" ]; then
        sleep $SLEEP_TIME
        NEW_SLEEP_TIME=`expr 2 * $SLEEP_TIME`
        if [ $NEW_SLEEP_TIME -ge 60 ]; then
            NEW_SLEEP_TIME=60
        fi
        initctl set-env SLEEP_TIME=$NEW_SLEEP_TIME
    fi
end script
share|improve this answer
1  
If you use respawn without arguments, the default it to retry up to ten times in a five minute window. –  Jamie Cockburn Feb 2 at 10:10
1  
How would you make sure this sleep didn't affect a legitimate stop of the job? ie, I stopped the job through initctl stop jobname - I wouldn't want it to have to wait through the sleep. –  Ashley Steel Feb 28 at 18:53
1  
The new code works perfectly. Thanks! I had to use $UPSTART_JOB instead of $serviceName. –  kichik Mar 12 at 18:56
1  
Excellent kichik. I've changed it to $UPSTART_JOB, per your comment. –  Roger Dueck Mar 12 at 19:02

I ended up putting a start in a cronjob. If the service is running, it has no effect. If it's not running, it starts the service.

share|improve this answer

You want respawn limit <times> <period> - although this would not provide the exponential behavior you are looking for, it probably would do for most use cases. You might try using very large values for times and period to approximate what you try to achieve. See the man 5 init's section on respawn limit for reference.

share|improve this answer
3  
The period is the period in which respawns are counted, not a delay between respawns. –  chrisdew Jan 28 '13 at 10:25

As already mentioned, use respawn to trigger the respawn.

However, the Upstart Cookbook coverage on respawn-limit says that you'll need to specify respawn limit unlimited to have continual retry behaviour.

By default it will retry as long as the process doesn't respawn more than 5 times in 10 seconds.

I would therefore suggest:

respawn
respawn limit unlimited
post-stop <script to back-off or constant delay>
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.