Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I want Upstart to do two things:

  1. stop trying to respawn a failed process so fast
  2. never give up trying to respawn

In an ideal world, upstart would try to restart a dead process after 1s, then double that delay on each attempt, until it reached an hour.

Is something like this possible?

share|improve this question
    
never give up trying to respawn remains unanswered. anyone? – vemv Oct 30 '15 at 1:22
up vote 20 down vote accepted
+50

The Upstart Cookbook recommends a post-stop delay (http://upstart.ubuntu.com/cookbook/#delay-respawn-of-a-job). Use the respawn stanza without arguments and it will continue trying forever:

respawn
post-stop exec sleep 5

(I got this from this Ask Ubuntu question)

To add the exponential delay part, I'd try working with an environment variable in the post-stop script, I think something like:

env SLEEP_TIME=1
post-stop script
    sleep $SLEEP_TIME
    NEW_SLEEP_TIME=`expr 2 \* $SLEEP_TIME`
    if [ $NEW_SLEEP_TIME -ge 60 ]; then
        NEW_SLEEP_TIME=60
    fi
    initctl set-env SLEEP_TIME=$NEW_SLEEP_TIME
end script

** EDIT **

To apply the delay only when respawning, avoiding the delay on a real stop, use the following, which checks whether the current goal is "stop" or not:

env SLEEP_TIME=1
post-stop script
    goal=`initctl status $UPSTART_JOB | awk '{print $2}' | cut -d '/' -f 1`
    if [ $goal != "stop" ]; then
        sleep $SLEEP_TIME
        NEW_SLEEP_TIME=`expr 2 \* $SLEEP_TIME`
        if [ $NEW_SLEEP_TIME -ge 60 ]; then
            NEW_SLEEP_TIME=60
        fi
        initctl set-env SLEEP_TIME=$NEW_SLEEP_TIME
    fi
end script
share|improve this answer
1  
If you use respawn without arguments, the default it to retry up to ten times in a five minute window. – Jamie Cockburn Feb 2 '15 at 10:10
2  
How would you make sure this sleep didn't affect a legitimate stop of the job? ie, I stopped the job through initctl stop jobname - I wouldn't want it to have to wait through the sleep. – Ashley Steel Feb 28 '15 at 18:53
1  
The new code works perfectly. Thanks! I had to use $UPSTART_JOB instead of $serviceName. – kichik Mar 12 '15 at 18:56
1  
Excellent kichik. I've changed it to $UPSTART_JOB, per your comment. – Roger Dueck Mar 12 '15 at 19:02

I ended up putting a start in a cronjob. If the service is running, it has no effect. If it's not running, it starts the service.

share|improve this answer
    
So janky and so elegant! <3 – pkoch Apr 26 at 0:26

As already mentioned, use respawn to trigger the respawn.

However, the Upstart Cookbook coverage on respawn-limit says that you'll need to specify respawn limit unlimited to have continual retry behaviour.

By default it will retry as long as the process doesn't respawn more than 5 times in 10 seconds.

I would therefore suggest:

respawn
respawn limit unlimited
post-stop <script to back-off or constant delay>
share|improve this answer

You want respawn limit <times> <period> - although this would not provide the exponential behavior you are looking for, it probably would do for most use cases. You might try using very large values for times and period to approximate what you try to achieve. See the man 5 init's section on respawn limit for reference.

share|improve this answer
4  
The period is the period in which respawns are counted, not a delay between respawns. – chrisdew Jan 28 '13 at 10:25

Others have answered the question for respawn and respawn limit stanzas, but I would like to add my own solution for the post-stop script that controls the delay between restarting.

The biggest problem with the solution proposed by Roger Dueck is that the delay causes 'restart jobName' to hang until the sleep is completed.

My addition checks to see if there is a restart in progress before determining whether or not to sleep.

respawn
respawn limit unlimited

post-stop script
    goal=`initctl status $UPSTART_JOB | awk '{print $2}' | cut -d '/' -f 1`
    if [[ $goal != "stop" ]]; then
            if ! ps aux | grep [r]estart | grep $UPSTART_JOB; then
                    sleep 60
            fi
    fi
end script
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.