Techniques to Monitor cron tasks?

Question

Are there good techniques for monitoring cron tasks over a cluster?

We're starting to use cron to launch tasks at daily intervals. A few ideas for checking out information:

1. Add special application handling that logs information into some "network aware" place, like a DB
2. Build up a logfile system that transfers the cron log periodically to a central point for processing/querying (along with other possible log files)

I'm wondering if people have had success with doing things separately for cron versus other things, or, if the tasks were integrated into a different approach completely. I'm leaning towards #2, but I'd like to know what more experienced folk might try out.

Answer 1

In addition to the other answers:

• let the job write a timestamp to a file when it finishes along with the return value from the actual job
• propagate the return value back to the original caller

We use the first to make it easier for Nagios (Icinga) to check, e.g if the last written timestamp is older than n hours (plus whatever logic you need) - we know something went wrong.

Answer 2

My common approach is thus:

• Don't produce any stdout when your cron'ed application completes successfully.
• Don't pipe any output to /dev/null.
• Do produce meaningful stderr output when something goes wrong.
• Do set a $MAILTO address in the crontab to send that error output to the required team.

Answer 3

Healthchecks (https://github.com/healthchecks/healthchecks/) is a service & dashboard built exactly for monitoring cron jobs. It is being used in production, is maintained and accepts code contributions.

It works similarly as Cronitor, Dead Man's Snitch and friends: you set up your cron job to make a HTTP/HTTPS request to a special, unique URL just before it finishes. Healthchecks receives and logs these pings. It constantly checks if the pings arrive at the expected intervals. When it detects a problem, it sends you a notification. Supported notification methods are email, webhooks, Slack, Telegram, Discord, SMS, Pushover, Pusbullet, PagerDuty, PagerTree, HipChat, VictorOps, OpsGenie.

You can set this all up and host yourself, but, as with any web service, it takes some effort to set up the domain name, the certificate, configure the HTTP reverse proxy, set up database backups etc. A reasonably easy way to get running is to use this Heroku-adapted version: https://github.com/iphoting/healthchecks. I know of people who run this project themselves and use it to monitor hundreds of services.

Disclaimer: I'm the author, and I also run Healthchecks as a hosted service at https://healthchecks.io

Answer 4

In addition to the above:

• Do call "logger" along with writing to stderr when something goes wrong. Configure syslog to additionally forward to a central host, aka "loghost". (Logger will use "user.notice" facility by default, but you may change it.)

Answer 5

There are a couple of techniques you could use for monitoring cronjobs.

To receive alerts of cronjob failures:

• Use cron's standard MAILTO= function. If a cronjob produces output on STDERR, it will be mailed to the address you choose.
• To track and deal with cron mails, you can direct them into a ticket system.

The system you propose to log information into a "network aware" place sounds like syslog. syslog provides a simple method for creating logs, it normally manages files such as /var/log/messages. You can make basic customisations, such as choosing which files receive the log messages.

Syslog can be started in a network aware mode. For example, you can configure it so a slave can log to a master:

[root@slave ~]#  echo "hello world from slave" | logger -p local1.info

[root@master ~]# tail /var/log/myapp
Jun 29 13:07:01 192.168.1.2 logger: hello world from slave

For a Red Hat based distribution, an example configuration is as follows:

[root@slave ~]# cat /etc/syslog.conf | grep local1
local1.*                                                @192.168.1.3

[root@master ~]# cat /etc/sysconfig/syslog | grep SYSLOGD_OPTIONS
SYSLOGD_OPTIONS="-m 0 -r"

[root@master ~]# cat /etc/syslog.conf | grep local
local1.* /var/log/myapp

(The first config line redirects local1.* log notices to @192.168.1.3 ("master"). The second SYSLOGD_OPIONS line's -r flag turns on network support. Lastly, the third config line directs local1.* messages received on "master" into a file).

The syslog approach is better for only logging errors/information. Log files have less visibility than e-mail, so you probably won't look at the logs unless something has gone wrong.

If you choose to go the syslog style route, also consider syslog-ng: http://freshmeat.net/projects/syslog-ng/.

Of course, you can get the best of both techniques by using both. For example, syslog'ing both failures and successes, and just mailing for failures.

Answer 6

I posted a similar answer to a question on StackOverflow(https://stackoverflow.com/questions/21025495/system-for-monitoring-cron-jobs-and-automated-tasks)

Cronitor (https://cronitor.io) was a tool I built exactly for this purpose. It basically boils down to being a tracking beacon that uses http requests as the pings.

However, one of the needs that the OP mentions in his comment is needing to be informed when a job starts taking too long to run.

I had this same need, and found that similar tools didn't easily support this type of monitoring. Cronitor solves this by allowing you to optionally trigger a begin event and an end event in order to keep track of duration.

Duration tracking was a must have for me because I had a cronjob that was scheduled every hour, but over time started taking over an hour to run. Hope you find it useful!

Answer 7

It's still under quite heavy development at the time of me writing this but I'd encourage to take a look at https://github.com/jamesrwhite/minicron. It was developed to solve the problems you describe. With a slight modification to the command you run it can record the output and exit status of jobs and sends that data back to a central server in realtime and can send alerts via email, SMS and PagerDuty when a job fails (exit status > 0) or doesn't execute when it should.

Disclaimer: I'm the developer working on it.

Answer 8

I have created Power Cron after these precise needs. I needed a centralized view over my cron jobs, and a notion of dependency between the jobs of different cluster members.

I also needed more information than what I could find in the logs, and added job profiling.

Answer 9

This looks like a classic use case for AlertGrid.

It doesn't require installation, all you need to do to take benefits from this tool is to:

1. send Signal to AlertGrid each time your cron job finished it's work (this can be done by extremaly simple API, signal is just a HTTP request). You can also send some parameters like execution_time!
2. set up notification rules like folllowing:

if my_job didn't respond in X minutes (hours in your case) -> send SMS to admin

or

if execution_time > 60 seconds -> send email to interested people

Actually that's all. You can manage notification rules using nice visual editor. You don't have to modify source code or some configuration files if something changed. It's centralized solution, so you can benefit from managing rules from a single place.

Hope this helps someone. There is a free account provided so you can test and use AlertGrid if you are interested. I am one of the AlertGrid team members - feel free to ask if you have some questions.

Answer 10

Your cron jobs are already logged via syslog. That data can be sent to a central server using syslogd, another standard service.

http://www.debuntu.org/how-to-remote-syslog-logging-on-debian-and-ubuntu/ has details on how to set this up.

Answer 11

i use http://cronrat.com just append && curl "...your cronrat url " to your cron jobs. The best feature i like is that you dont need to set up anything after you create initial account. Each alert is up and running the minute you use it. therefore i can use any automated tools to start my jobs that dont exists yet, unlike on some services where i need to set up job first.

Answer 12

We built PushMon, http://www.pushmon.com, for this. Say your daily job runs at 3 AM and normally ends at 4 AM. You can setup a PushMon schedule of "by 4:00 AM every day". Or a little more advanced schedule like "by 4:00 AM every day within 1 hour". All you need to do is "ping" the PushMon URL every time your job runs, and it will alert you of missing pings. If you know for sure an error has occurred, like when you catch an exception you cannot handle, you can use the on- demand alert feature.