23

I want Upstart to do two things:

  1. stop trying to respawn a failed process so fast
  2. never give up trying to respawn

In an ideal world, upstart would try to restart a dead process after 1s, then double that delay on each attempt, until it reached an hour.

Is something like this possible?

1
  • never give up trying to respawn remains unanswered. anyone?
    – deprecated
    Oct 30, 2015 at 1:22

6 Answers 6

29
+50

The Upstart Cookbook recommends a post-stop delay (http://upstart.ubuntu.com/cookbook/#delay-respawn-of-a-job). Use the respawn stanza without arguments and it will continue trying forever:

respawn
post-stop exec sleep 5

(I got this from this Ask Ubuntu question)

To add the exponential delay part, I'd try working with an environment variable in the post-stop script, I think something like:

env SLEEP_TIME=1
post-stop script
    sleep $SLEEP_TIME
    NEW_SLEEP_TIME=`expr 2 \* $SLEEP_TIME`
    if [ $NEW_SLEEP_TIME -ge 60 ]; then
        NEW_SLEEP_TIME=60
    fi
    initctl set-env SLEEP_TIME=$NEW_SLEEP_TIME
end script

** EDIT **

To apply the delay only when respawning, avoiding the delay on a real stop, use the following, which checks whether the current goal is "stop" or not:

env SLEEP_TIME=1
post-stop script
    goal=`initctl status $UPSTART_JOB | awk '{print $2}' | cut -d '/' -f 1`
    if [ $goal != "stop" ]; then
        sleep $SLEEP_TIME
        NEW_SLEEP_TIME=`expr 2 \* $SLEEP_TIME`
        if [ $NEW_SLEEP_TIME -ge 60 ]; then
            NEW_SLEEP_TIME=60
        fi
        initctl set-env SLEEP_TIME=$NEW_SLEEP_TIME
    fi
end script
4
  • 1
    If you use respawn without arguments, the default it to retry up to ten times in a five minute window. Feb 2, 2015 at 10:10
  • 3
    The problem with this for a production system is that once you reach the max (60s) it will always take 60secs even if the system is back to healthy. Maybe there could be post-start to reset it to 1. Oct 17, 2016 at 18:38
  • 2
    @JamieCockburn The default interval is not 5 minutes, it's 5 seconds.
    – Zitrax
    Jan 10, 2017 at 13:04
  • 1
    This almost worked for me - but the set-env trick hit "initctl: Not permissible to modify PID 1 job environment" . Instead I had to resort to storing the sleep value in /tmp/$UPSTART_JOB and then sourcing it back in
    – Goblinhack
    Feb 3, 2017 at 15:43
5

As already mentioned, use respawn to trigger the respawn.

However, the Upstart Cookbook coverage on respawn-limit says that you'll need to specify respawn limit unlimited to have continual retry behaviour.

By default it will retry as long as the process doesn't respawn more than 10 times in 5 seconds.

I would therefore suggest:

respawn
respawn limit unlimited
post-stop <script to back-off or constant delay>
4

I ended up putting a start in a cronjob. If the service is running, it has no effect. If it's not running, it starts the service.

1
  • 3
    So janky and so elegant! <3
    – pkoch
    Apr 26, 2016 at 0:26
3

I have done an improvement to Roger answer. Typically you want to backoff when there is a problem in the underlying software causing it to crash a lot in a short period of time but once the system has recovered you want to reset the backoff time. In Roger's version the service will sleep for 60 seconds always, even for single and isolated crashes after 7 crashes.

#The initial delay.
env INITIAL_SLEEP_TIME=1

#The current delay.
env CURRENT_SLEEP_TIME=1

#The maximum delay
env MAX_SLEEP_TIME=60

#The unix timestamp of the last crash.
env LAST_CRASH=0

#The number of seconds without any crash 
#to consider the service healthy and reset the backoff.
env HEALTHY_TRESHOLD=180

post-stop script
  exec >> /var/log/auth0.log 2>&1
  echo "`date`: stopped $UPSTART_JOB"
  goal=`initctl status $UPSTART_JOB | awk '{print $2}' | cut -d '/' -f 1`
  if [ $goal != "stop" ]; then
    CRASH_TIMESTAMP=$(date +%s)

    if [ $LAST_CRASH -ne 0 ]; then
      SECS_SINCE_LAST_CRASH=`expr $CRASH_TIMESTAMP - $LAST_CRASH`
      if [ $SECS_SINCE_LAST_CRASH -ge $HEALTHY_TRESHOLD ]; then
        echo "resetting backoff"
        CURRENT_SLEEP_TIME=$INITIAL_SLEEP_TIME
      fi
    fi

    echo "backoff for $CURRENT_SLEEP_TIME"
    sleep $CURRENT_SLEEP_TIME

    NEW_SLEEP_TIME=`expr 2 \* $CURRENT_SLEEP_TIME`
    if [ $NEW_SLEEP_TIME -ge $MAX_SLEEP_TIME ]; then
      NEW_SLEEP_TIME=$MAX_SLEEP_TIME
    fi

    initctl set-env CURRENT_SLEEP_TIME=$NEW_SLEEP_TIME
    initctl set-env LAST_CRASH=$CRASH_TIMESTAMP
  fi
end script
1

You want respawn limit <times> <period> - although this would not provide the exponential behavior you are looking for, it probably would do for most use cases. You might try using very large values for times and period to approximate what you try to achieve. See the man 5 init's section on respawn limit for reference.

2
  • 6
    The period is the period in which respawns are counted, not a delay between respawns.
    – fadedbee
    Jan 28, 2013 at 10:25
  • 1
    Which I assume would mean that even if you used respawn limit 10 3600 the 10 tries would likely be used up immediately - since by default there is no delay.
    – Zitrax
    Jan 10, 2017 at 13:33
0

Others have answered the question for respawn and respawn limit stanzas, but I would like to add my own solution for the post-stop script that controls the delay between restarting.

The biggest problem with the solution proposed by Roger Dueck is that the delay causes 'restart jobName' to hang until the sleep is completed.

My addition checks to see if there is a restart in progress before determining whether or not to sleep.

respawn
respawn limit unlimited

post-stop script
    goal=`initctl status $UPSTART_JOB | awk '{print $2}' | cut -d '/' -f 1`
    if [[ $goal != "stop" ]]; then
            if ! ps aux | grep [r]estart | grep $UPSTART_JOB; then
                    sleep 60
            fi
    fi
end script

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .