Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I run a job every minute to reindex my site's content.

Today, the search engine died, and when I logged in there were hundreds of orphan processes that had been started by cron.

Is there another way using some kind of existing software that will let me execute a job every minute, but that won't launch another instance if that job doesn't return (i.e. because the search engine process has failed)?

share|improve this question
4  
cron is most likely doing exactly what you are telling to. I suggest rewriting the job intelligently instead. –  gparent Apr 4 '12 at 16:28

4 Answers 4

up vote 23 down vote accepted

The problem isn't really with cron - it's with your job.

You will need to have your job interact with a lock of some description. The easiest way to do this is have it attempt to create a directory and if successful continue, if not exit. When your job has finished and exits it should remove the directory ready for the next run. Here's a script to illustrate.

#!/bin/bash

function cleanup {
    echo "Cleanup"
    rmdir /tmp/myjob.lck
}

mkdir /tmp/myjob.lck ||  exit 1
trap cleanup EXIT
echo 'Job Running'
sleep  60
exit 0

Run this in one terminal then before 60 seconds is up run it in another terminal it will exit with status 1. Once the first process exits you can run it from the second terminal ...

EDIT:

As I just learned about flock I thought I'd update this answer. flock(1) may be easier to use. In this case flock -n would seem appropriate e.g.

* * * * * /usr/bin/flock -n /tmp/myAppLock.lck /path/to/your/job   

Would run your job every minute but would fail if flock could not obtain a lock on the file.

share|improve this answer
2  
Stupid question maybe, but are there any advantages to using a directory specifically rather than a regular file? –  gparent Apr 4 '12 at 16:34
9  
Using a regular file requires several operations, check to see if it exists, if it doesn't then create it. This leaves a window of opportunity for another process to create the file - messy. The mkdir is an atomic operation it either works and you get the 'lock' or it doesn't as another process already has it. –  Iain Apr 4 '12 at 16:42
3  
Good to know, thanks! –  gparent Apr 4 '12 at 16:45
    
Makes sense. Good thinking on the lock directory too. Thanks –  John Apr 5 '12 at 9:44

Another way to quickly set this up is to have a shell script start when the machine starts up (cron can do this with '@reboot /path/to/my/script.sh',. then restart the cron to start it up) with something like this in it.

#!/bin/sh
/opt/bin/run-site-index
sleep 60
exec $0

The script keeps running, and is you have only started one - thats how many could be running at once - no more than that. Some smarts in there can also check if the indexer is running, and if not, restart, or otherwise try to fix/notify someone of the issue.

share|improve this answer

Instead of using cron for this I would build your job more as a service that runs in a loop and sleeps for 60 seconds as the last step, or perhaps sleeps more often for smaller intervals at various points during the process to help spread out the load more evenly.

share|improve this answer
1  
This would neither fix the issue nor be an improvement from cron. –  gparent Apr 4 '12 at 16:29
    
It would fix the issue, because then there's only one process that runs ever. It would sidestep cron completely. –  Joel Coel Apr 4 '12 at 16:30
    
It doesn't fix the issue if the 'service' doesn't look if the search engine is running. His script/job's logic is the issue. EDIT: Actually, you're somewhat right, it would hide the issue in an ugly way. –  gparent Apr 4 '12 at 16:31

One way would be to have your reindex script create a lock file so that it can check to see if there's already an instance of the script running. You can also add in some exception handling to see if the search engine is up and running.

A more involved alternative would be to use some sort of task queuer like Resque and Resque-scheduler:

https://github.com/blog/542-introducing-resque

https://github.com/bvandenbos/resque-scheduler#readme

There's also Qu and Sidekiq:

https://github.com/bkeepers/qu

https://github.com/mperham/sidekiq

Yes, that's all Ruby orientated, but you can look for "things like resque" in the language of your choice.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.