4

I realize that when any cron has output it will email that output... what I'm trying to determine is that if I have a script scheduled for 3 AM on Tuesday and for whatever reason it either throws an error or fails to run, I'd like to know...

I'm thinking right now of setting up a database table that stores last run timestamps for each cron command and we get weekly report for the cron commands. Or possibly storing in the database when it should run and when it last ran, if there's a problem it'd email us.

The "emailing" would be done by our internal systems where our employees are consistently logging in so it wouldn't be based off of cron itself.

Is there a better solution?

1
  • 1
    Why does your script throws an error or fails to run? Is it because of code errors or the script depends on things that you can't control? Aug 28, 2013 at 19:04

8 Answers 8

4

I think monitoring syslog would be the easiest solution.

Have your syslogs forward to your monitoring system, and then set up alerts within your monitoring system.

I've also set up custom SNMP MIBs in the past, which you could put the timestamp of the last time that particular cronjob has run. Then some external system could monitor that snmp MIB for a timestamp older than 24 hours.

3

Your solution is workable, but it reinvents some wheels that you probably don't need to.

First, you should really have some type of monitoring service. I tend to use nagios, but there's a ton out there. Choose one of those systems and have it monitor your cron daemon.

Then write a plugin that will utilize the wrappers that voretaq7 mentioned. You will have an alert if the cronjob fails and if crond also fails.

The reason I suggest this is that you'll have all your monitoring in one place. You'll eventually have to have a site wide monitoring system and it makes more sense to put your effort into that rather than having a scattered series of monitoring systems in place.

3
  • 1
    Nagios looks so bad... is there an open source/free alternative that looks professional?
    – Ben
    Jan 28, 2011 at 20:37
  • As I mentioned before, there are tons of others, a quick search on freshmeat or sourceforge should turn up quite a few.
    – D.F.
    Jan 28, 2011 at 20:55
  • OpenNMS--> opennms.org and here is how it works with syslog--> opennms.org/wiki/Syslogd Jan 31, 2011 at 16:27
3

I built a simple tool for this type of monitoring - https://cronitor.io

It allows you to set both intervals (every 24 hours) and durations (greater than 10 minutes, less than 2 minutes, etc), and then receive email/SMS alerts if your cron job (or any other automated task) doesn't run according to the rules you defined.

The tool is free for individual monitors and paid plans are available for those with multiple monitoring needs.

2

Your solution sounds OK based on your environment, but it might be a bit of overkill (unless you need to be able to audit the history of this job over the long term, in which case the database bit might make sense).

Another option to consider is simply wrapping your cron jobs in a check script (if the cron job exits with an error status (!=0) send an email, or generate output and let cron send the email for you).

1
  • 1
    That sounds good, but if say, crontab isn't running the cron won't run and thus not execute the shell script either.
    – Ben
    Jan 28, 2011 at 16:39
2

Your solution sounds a little more complicated than I think it needs to be.

Start with reviewing and/or monitoring /var/log/cron.log (or wherever your cron logs are going). cron does a good job of logging every command that it runs, along with errors. If you want to know what happened, that's the place to look. If you're worried about cron dying, you can setup a cron'ed heartbeat that just logs every 5 minutes, and if you don't see the heartbeat, send some sort of alert. If you really feel like you need a second tool keeping an eye on cron, there's a perl package (Schedule::Cron) that you could use to regularly check your heartbeat. If you're that worried about the local machine's reliability, you can also send the logs to a second machine for monitoring/processing/alerting/etc.

Alternately, you could just use some sort of system monitoring tool (SNMP, Nagios, Hobbit/BigSister, etc) to externally monitor that the cron process is running. You do monitor your systems' health, right?

Although, if you're really that worried about cron dying, you might want to look into rebuilding or replacing your machine. cron should be pretty reliable, and if it's failing, it's probably a symptom of a bigger problem.

3
  • System health is not monitored currently. I'm looking for a solution that will bring the information to us, rather than us having to go looking for it.
    – Ben
    Jan 28, 2011 at 17:49
  • Monitoring /var/log/cron.log doesn't have to be a manual process. It's pretty easy to setup a script to watch the log and e-mail (or otherwise alert you) on certain actions. There are lots of tools out there to do basic log monitoring (swatch is a commonly used one). Jan 28, 2011 at 18:10
  • Regarding the last paragraph: We had crons fail because the user account met its process limit. So, there definitely are scenarios (well, at least one :) ) in which cron jobs fail even when cron is working fine.
    – rinogo
    Aug 23, 2019 at 0:59
2

I have a dealt with a similar requirement:

The script run by cron sends it's output to the logger command. logger sends a syslog message to the Local4 facility, which is handled by rsyslog. The local4.* is then sent to a remote Syslog listener - in my case, a Splunk instance. Splunk has a saved search which fires email alerts if the events do not happen within the expected time window. In addition to the alerts, Splunk also gives me a nice searchable historical timeline of the events.

2

You can use PushMon and create a URL with a "by 3:30 AM every Tue" schedule. Then "ping" the PushMon URL when your script runs successfully. If the PushMon URL doesn't get called because the machine is off, or cron failed to run (it happens), or your script fails, PushMon will alert you by 3:30 AM. You can get alerted by email, SMS, phone, IM or Twitter, and the service is free.

Disclaimer: I am associated with PushMon.

0

Try healthchecks.io, it is an excellent free and open source solution. You can even host it if you want.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .