0

I have a queue script that finds new jobs from the db every second and handles them. Sometimes it doesn't do anything for 10 hours, and sometimes it receives 1000 new jobs in 3 min. The queue works fine, mostly.

It needs restarting though. (I'm not entirely sure why. I think other services the jobs talk to don't like connections being open for a long time. Restarting the queue, resets all connections. Maybe that's not why.) And sometimes the queue script just dies. Maybe memory error, I can't pinpoint it.

There's 2 ways to restart that I'm both fine with, (but it has to be automatic):

  1. Explicitly restart it every 24h: ctrl C + ./queue.sh
  2. Wait until it dies, and start it again

I'm not sure about either... The queue runs in a screen so I can follow output when I want. How can one command listen for another command to finish and restart it, without being a daemon?

I can't install anything. It's a crappy Redhat server that I don't have decent admin access to.

I've thought about creating a cronjob that fires every 24h and kills itself after 24h, but that just sounds so wrong... I can't use the cronjob for the queue, because new jobs must be executed almost-instantly.

3

You can run ./queue.sh as a child of a monitor script, like this:

1. #!/bin/bash
2. while true; do
3.     ./queue.sh & q_pid="${!}"
4.     ( sleep 86400 ; kill "${q_pid}" ) & s_pid="${!}"
5.     wait "${q_pid}"
6.     kill "${s_pid}"
7.     wait "${s_pid}"
8. done

How it works:

  1. The monitor script runs a ./queue.sh process in background (line 3) and a subprocess that sleeps for one day and kills ./queue.sh (line 4). Then, it waits ./queue.sh to finish (line 5).

  2. If ./queue.sh ends prematurely, the monitor kills the sleep process (line 6), so it will not kill an innocent process with the same PID in the future. The wait command at line 7 prevents the sleep process from being a zombie.

  3. If ./queue.sh execution lasts for more than 24 hours, it is forcefully finished when the sleep process ends (line 4).

  • Won't the sleep only start after ./queue.sh finishes? I like the entire script being in 1 file though, without cron. – Rudie Mar 6 '16 at 23:04
  • The & character means "run the ./queue.sh in background and continue". Then, the wait command blocks the loop after the background launching of sleep command. – Anderson Medeiros Gomes Mar 6 '16 at 23:24
  • I want to capture the output, using tee, and then mail it, every day. Where would I add the mail command? After line 5 or 7? – Rudie Mar 7 '16 at 0:40
  • I suggest you to add the mail command after line 7. However, beware of modifying line 3. For example, if you write ( ./queue.sh 2>&1 ) | tee /path/to/temporary/file & q_pid="${!}" and the queue processing hangs, the monitor script will not kill the ./queue.sh process. – Anderson Medeiros Gomes Mar 7 '16 at 1:06
1

queue.sh must be running some kind of process. You can modify it such that it loops - something like this (not necessary valid bash below, just some pseudo code):

  do
      processqueue
  loop

Then write a cronjob that kills processqueue process at 1AM or whatever time you like. Process will die, and the loop will restart it.

  • So there's 2 processes, and the outer will restart the inner after killing the inner? So the outer process will run forever and ever? – Rudie Mar 6 '16 at 20:27
  • @Rudie - that's the idea, yes. Not knowing more specifics as to your script, it's hard to be more precise. But yes, the shell script you run would start the actual process that does the actual work so that if it stops, it restarts it. There are other tools to do that obviously, such as respawn superuser.com/questions/507576/… and supervise stackoverflow.com/questions/9351026/… – ETL Mar 6 '16 at 21:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.