Auto-restart commandline process in screen every day

Question

I have a queue script that finds new jobs from the db every second and handles them. Sometimes it doesn't do anything for 10 hours, and sometimes it receives 1000 new jobs in 3 min. The queue works fine, mostly.

It needs restarting though. (I'm not entirely sure why. I think other services the jobs talk to don't like connections being open for a long time. Restarting the queue, resets all connections. Maybe that's not why.) And sometimes the queue script just dies. Maybe memory error, I can't pinpoint it.

There's 2 ways to restart that I'm both fine with, (but it has to be automatic):

1. Explicitly restart it every 24h: ctrl C + ./queue.sh
2. Wait until it dies, and start it again

I'm not sure about either... The queue runs in a screen so I can follow output when I want. How can one command listen for another command to finish and restart it, without being a daemon?

I can't install anything. It's a crappy Redhat server that I don't have decent admin access to.

I've thought about creating a cronjob that fires every 24h and kills itself after 24h, but that just sounds so wrong... I can't use the cronjob for the queue, because new jobs must be executed almost-instantly.

Answer 1

You can run ./queue.sh as a child of a monitor script, like this:

1. #!/bin/bash
2. while true; do
3.     ./queue.sh & q_pid="${!}"
4.     ( sleep 86400 ; kill "${q_pid}" ) & s_pid="${!}"
5.     wait "${q_pid}"
6.     kill "${s_pid}"
7.     wait "${s_pid}"
8. done

How it works:

1. The monitor script runs a ./queue.sh process in background (line 3) and a subprocess that sleeps for one day and kills ./queue.sh (line 4). Then, it waits ./queue.sh to finish (line 5).

2. If ./queue.sh ends prematurely, the monitor kills the sleep process (line 6), so it will not kill an innocent process with the same PID in the future. The wait command at line 7 prevents the sleep process from being a zombie.

3. If ./queue.sh execution lasts for more than 24 hours, it is forcefully finished when the sleep process ends (line 4).

Answer 2

queue.sh must be running some kind of process. You can modify it such that it loops - something like this (not necessary valid bash below, just some pseudo code):

  do
      processqueue
  loop

Then write a cronjob that kills processqueue process at 1AM or whatever time you like. Process will die, and the loop will restart it.