Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

As per the title:

What's the best way of setting up a cron job to check that a long-running process is still going and if not, start it?

If I start a long-running process in cron, is it going to block? or does cron fork the process as an independent child?

Thanks!

share|improve this question
    
Im not a linux expert at all, but seeing as there's no other answers yet, I'm going to take a guess that you're going to need to google PS and GREP. As for how to combine the two, I've no idea. –  Mark Henderson Aug 20 '09 at 9:58
    
Or it could be TOP and GREP. As I said, not a *nix person ;) –  Mark Henderson Aug 20 '09 at 9:59
    
A pgrep would be a 'better' way to grep for a process –  Rory Aug 20 '09 at 10:59

5 Answers 5

up vote 5 down vote accepted

What's the best way of setting up a cron job to check that a long-running process is still going and if not, start it?

A simple approach is to have a straightforward script that checks whether the process is running or not, and then restart it when necessary.

(Sometimes it's best to actually validate that the process is running by means of a 'dummy transaction', e.g. to verify an SMTP process you might make a TCP port connection and check that it responds correctly.)

But do watch out for differences in your environment between you as an interactive user, and when cron(8) runs your script.

To answer the second bit of your question:

If I start a long-running process in cron, is it going to block? or does cron fork the process as an independent child?

cron(8) will fork to execute a cron job, but unless your script or process 'detaches', cron will maintain it as a child process until it exits (that's how cron is able to collect all output from stderr, and send it via email.)

But, I think you were thinking, could you actually run the long-running process from cron? If you do that, you need to make sure that it can only start one-copy of itself, and that it will quickly exit if it is already running.

Better solutions for keeping long-running processes running - if you're only worried about an exit or crash

  • if your process can be made to stay attached, use init(1) via inittab(5) and the 'respawn' option. Often even daemons have "no fork" options.
  • Or, if your OS doesn't have an inittab feature or you don't have access to it, use something like DJB's daemontools.
  • if you had the luxury of using Solaris 10, or OpenSolaris, you can use SMF. (This can even work with processes that do fork and detach.)
  • If it's your own code, you can write it to have a parent/child pair of processes, where the parent restarts the child whenever it receives a SIGCHLD.
share|improve this answer

A common idiom is for a long running process to have a pid file. Basically a file in /var/run or similar that just has the pid, or process id of the programme. When the programme starts it puts the file there, when it stops, it removes the file. You can easily check that the programme is there by seeing if that file is there.

This can also be used to see if the programme has crashed. If the file is there, but there is no process running with that pid, then the programme stopped, but didn't remove the pid file, i.e. it crashed. In this case, you can remove the pid file and restart the programme. However this isn't fool proof as sometimes the PID could get reused by a new process that started after your original one crashed.

share|improve this answer
    
+1 - Just be sure you check to see if the PID listed in the file you write into /var/run is "stale" by grepping the processs list for that PID and the program's name. –  Evan Anderson Aug 20 '09 at 10:05

Depending on how do you detect your process, the cronjob might look like

* * * * * pidof executable || /usr/local/bin/executable

provided that your executable presents itself as itself in process list. More sensible way would be to have a pidfile and use start-stop-daemon. Actually, it all depends on the process in question. A while ago I've also written a small process maintenance daemon dudki exactly for this purpose.

And no, cron won't block, but depending on the nature of your process you may want to background it, anyway.

share|improve this answer

By "long running" do you mean something that just takes a long time to run, or a daemon that should be running continually?

If you mean a process that takes a long time to complete, then ,as suggested, have the process create a pid file and look for that file at start up. If the file exists, spew a message and exit cleanly. Otherwise start up as normal.

If you mean a daemon (sometimes called a service), instead of using cron, you should take a look at ps-watcher, I use it to do the same thing.

From the website: "...it can be used to ensure that a daemon is running, or is not running too many times. It can also be used to determine when a process has consumed too many resources, perhaps due to a memory leak."

You just configure ps-watcher to look for your process. ps-watcher will check the process list and if it's not found, ps-watcher will start it for you.

share|improve this answer

monit was designed to solve this problem.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.