Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Are there good techniques for monitoring cron tasks over a cluster?

We're starting to use cron to launch tasks at daily intervals. A few ideas for checking out information:

  1. Add special application handling that logs information into some "network aware" place, like a DB
  2. Build up a logfile system that transfers the cron log periodically to a central point for processing/querying (along with other possible log files)

I'm wondering if people have had success with doing things separately for cron versus other things, or, if the tasks were integrated into a different approach completely. I'm leaning towards #2, but I'd like to know what more experienced folk might try out.

share|improve this question
    
is your concern that cronjobs are not running? or are you asking to monitor the 'status' for job run? –  ericslaw Jun 29 '09 at 12:04
    
Mostly, that they didn't fail. But some jobs take a long time, and we may want to grab information like "oops, this is taking too long". –  Tristan Juricek Jun 30 '09 at 8:43
add comment

11 Answers 11

up vote 10 down vote accepted

In addition to the other answers:

  • let the job write a timestamp to a file when it finishes along with the return value from the actual job
  • propagate the return value back to the original caller

We use the first to make it easier for Nagios (Icinga) to check, e.g if the last written timestamp is older than n hours (plus whatever logic you need) - we know something went wrong.

share|improve this answer
    
While I like everyone's answers - I learned a lot - I completely forgot about our Nagios monitoring. This is great for those long running tasks, what I'm really concerned about. Thanks. –  Tristan Juricek Jun 30 '09 at 8:52
add comment

I posted a similar answer to a question on StackOverflow(http://stackoverflow.com/questions/21025495/system-for-monitoring-cron-jobs-and-automated-tasks)

Cronitor (https://cronitor.io) was a tool I built exactly for this purpose. It basically boils down to being a tracking beacon that uses http requests as the pings.

However, one of the needs that the OP mentions in his comment is needing to be informed when a job starts taking too long to run.

I had this same need, and found that similar tools didn't easily support this type of monitoring. Cronitor solves this by allowing you to optionally trigger a begin event and an end event in order to keep track of duration.

Duration tracking was a must have for me because I had a cronjob that was scheduled every hour, but over time started taking over an hour to run. Hope you find it useful!

share|improve this answer
add comment

It's still under quite heavy development at the time of me writing this but I'd encourage to take a look at https://github.com/jamesrwhite/minicron. It was developed to solve the problems you describe. With a slight modification to the command you run it can record the output and exit status of jobs and sends that data back to a central server in realtime and can send alerts via email, SMS and PagerDuty when a job fails (exit status > 0) or doesn't execute when it should.

Disclaimer: I'm the developer working on it.

share|improve this answer
add comment

i use http://cronrat.com just append && curl "...your cronrat url " to your cron jobs. The best feature i like is that you dont need to set up anything after you create initial account. Each alert is up and running the minute you use it. therefore i can use any automated tools to start my jobs that dont exists yet, unlike on some services where i need to set up job first.

share|improve this answer
add comment

Your cron jobs are already logged via syslog. That data can be sent to a central server using syslogd, another standard service.

http://www.debuntu.org/how-to-remote-syslog-logging-on-debian-and-ubuntu/ has details on how to set this up.

share|improve this answer
add comment

You could also try out snooze: https://snooze.io . Comes with nice ruby integration.

share|improve this answer
add comment

We use Cronwatch to monitor if our crons are running at the correct intervals. Cronwatch can also check the output to ensure that the script completed successfully.

It also logs the output so you have an audit trail of logs.

share|improve this answer
add comment

This looks like a classic use case for AlertGrid.

It doesn't require installation, all you need to do to take benefits from this tool is to:

  1. send Signal to AlertGrid each time your cron job finished it's work (this can be done by extremaly simple API, signal is just a HTTP request). You can also send some parameters like execution_time!
  2. set up notification rules like folllowing:

if my_job didn't respond in X minutes (hours in your case) -> send SMS to admin

or

if execution_time > 60 seconds -> send email to interested people

Actually that's all. You can manage notification rules using nice visual editor. You don't have to modify source code or some configuration files if something changed. It's centralized solution, so you can benefit from managing rules from a single place.

Hope this helps someone. There is a free account provided so you can test and use AlertGrid if you are interested. I am one of the AlertGrid team members - feel free to ask if you have some questions.

share|improve this answer
add comment

There are a couple of techniques you could use for monitoring cronjobs.

To receive alerts of cronjob failures:

  • Use cron's standard MAILTO= function. If a cronjob produces output on STDERR, it will be mailed to the address you choose.
  • To track and deal with cron mails, you can direct them into a ticket system.

The system you propose to log information into a "network aware" place sounds like syslog. syslog provides a simple method for creating logs, it normally manages files such as /var/log/messages. You can make basic customisations, such as choosing which files receive the log messages.

Syslog can be started in a network aware mode. For example, you can configure it so a slave can log to a master:

[root@slave ~]#  echo "hello world from slave" | logger -p local1.info

[root@master ~]# tail /var/log/myapp
Jun 29 13:07:01 192.168.1.2 logger: hello world from slave

For a Red Hat based distribution, an example configuration is as follows:

[root@slave ~]# cat /etc/syslog.conf | grep local1
local1.*                                                @192.168.1.3

[root@master ~]# cat /etc/sysconfig/syslog | grep SYSLOGD_OPTIONS
SYSLOGD_OPTIONS="-m 0 -r"

[root@master ~]# cat /etc/syslog.conf | grep local
local1.* /var/log/myapp

(The first config line redirects local1.* log notices to @192.168.1.3 ("master"). The second SYSLOGD_OPIONS line's -r flag turns on network support. Lastly, the third config line directs local1.* messages received on "master" into a file).

The syslog approach is better for only logging errors/information. Log files have less visibility than e-mail, so you probably won't look at the logs unless something has gone wrong.

If you choose to go the syslog style route, also consider syslog-ng: http://freshmeat.net/projects/syslog-ng/.

Of course, you can get the best of both techniques by using both. For example, syslog'ing both failures and successes, and just mailing for failures.

share|improve this answer
    
Thanks for the answer -> I'm a programmer, which makes me a bit of a sysadmin newbie. I wasn't even aware of the network capabilities of syslog. –  Tristan Juricek Jun 30 '09 at 8:47
add comment

In addition to the above:

  • Do call "logger" along with writing to stderr when something goes wrong. Configure syslog to additionally forward to a central host, aka "loghost". (Logger will use "user.notice" facility by default, but you may change it.)
share|improve this answer
    
I like this idea.... though crond already logs to syslog (perhaps via config param) so use of logger isn't strictly required for this approach. –  ericslaw Jun 29 '09 at 12:03
add comment

My common approach is thus:

  • Don't produce any stdout when your cron'ed application completes successfully.
  • Don't pipe any output to /dev/null.
  • Do produce meaningful stderr output when something goes wrong.
  • Do set a $MAILTO address in the crontab to send that error output to the required team.
share|improve this answer
    
And if one really has to pipe output to /dev/null at least add || echo "service $service is FUBAR" to the command line... –  Hubert Kario Jul 16 '12 at 13:45
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.