Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I realize that when any cron has output it will email that output... what I'm trying to determine is that if I have a script scheduled for 3 AM on Tuesday and for whatever reason it either throws an error or fails to run, I'd like to know...

I'm thinking right now of setting up a database table that stores last run timestamps for each cron command and we get weekly report for the cron commands. Or possibly storing in the database when it should run and when it last ran, if there's a problem it'd email us.

The "emailing" would be done by our internal systems where our employees are consistently logging in so it wouldn't be based off of cron itself.

Is there a better solution?

share|improve this question
    
Why does your script throws an error or fails to run? Is it because of code errors or the script depends on things that you can't control? –  nacholibre Aug 28 '13 at 19:04
add comment

7 Answers 7

I think monitoring syslog would be the easiest solution.

Have your syslogs forward to your monitoring system, and then set up alerts within your monitoring system.

I've also set up custom SNMP MIBs in the past, which you could put the timestamp of the last time that particular cronjob has run. Then some external system could monitor that snmp MIB for a timestamp older than 24 hours.

share|improve this answer
add comment

Your solution sounds OK based on your environment, but it might be a bit of overkill (unless you need to be able to audit the history of this job over the long term, in which case the database bit might make sense).

Another option to consider is simply wrapping your cron jobs in a check script (if the cron job exits with an error status (!=0) send an email, or generate output and let cron send the email for you).

share|improve this answer
    
That sounds good, but if say, crontab isn't running the cron won't run and thus not execute the shell script either. –  Webnet Jan 28 '11 at 16:39
add comment

Your solution sounds a little more complicated than I think it needs to be.

Start with reviewing and/or monitoring /var/log/cron.log (or wherever your cron logs are going). cron does a good job of logging every command that it runs, along with errors. If you want to know what happened, that's the place to look. If you're worried about cron dying, you can setup a cron'ed heartbeat that just logs every 5 minutes, and if you don't see the heartbeat, send some sort of alert. If you really feel like you need a second tool keeping an eye on cron, there's a perl package (Schedule::Cron) that you could use to regularly check your heartbeat. If you're that worried about the local machine's reliability, you can also send the logs to a second machine for monitoring/processing/alerting/etc.

Alternately, you could just use some sort of system monitoring tool (SNMP, Nagios, Hobbit/BigSister, etc) to externally monitor that the cron process is running. You do monitor your systems' health, right?

Although, if you're really that worried about cron dying, you might want to look into rebuilding or replacing your machine. cron should be pretty reliable, and if it's failing, it's probably a symptom of a bigger problem.

share|improve this answer
    
System health is not monitored currently. I'm looking for a solution that will bring the information to us, rather than us having to go looking for it. –  Webnet Jan 28 '11 at 17:49
    
Monitoring /var/log/cron.log doesn't have to be a manual process. It's pretty easy to setup a script to watch the log and e-mail (or otherwise alert you) on certain actions. There are lots of tools out there to do basic log monitoring (swatch is a commonly used one). –  Christopher Cashell Jan 28 '11 at 18:10
add comment

I have a dealt with a similar requirement:

The script run by cron sends it's output to the logger command. logger sends a syslog message to the Local4 facility, which is handled by rsyslog. The local4.* is then sent to a remote Syslog listener - in my case, a Splunk instance. Splunk has a saved search which fires email alerts if the events do not happen within the expected time window. In addition to the alerts, Splunk also gives me a nice searchable historical timeline of the events.

share|improve this answer
add comment

Your solution is workable, but it reinvents some wheels that you probably don't need to.

First, you should really have some type of monitoring service. I tend to use nagios, but there's a ton out there. Choose one of those systems and have it monitor your cron daemon.

Then write a plugin that will utilize the wrappers that voretaq7 mentioned. You will have an alert if the cronjob fails and if crond also fails.

The reason I suggest this is that you'll have all your monitoring in one place. You'll eventually have to have a site wide monitoring system and it makes more sense to put your effort into that rather than having a scattered series of monitoring systems in place.

share|improve this answer
    
Nagios looks so bad... is there an open source/free alternative that looks professional? –  Webnet Jan 28 '11 at 20:37
    
As I mentioned before, there are tons of others, a quick search on freshmeat or sourceforge should turn up quite a few. –  D.F. Jan 28 '11 at 20:55
    
OpenNMS--> opennms.org and here is how it works with syslog--> opennms.org/wiki/Syslogd –  JakeRobinson Jan 31 '11 at 16:27
add comment

You can use PushMon and create a URL with a "by 3:30 AM every Tue" schedule. Then "ping" the PushMon URL when your script runs successfully. If the PushMon URL doesn't get called because the machine is off, or cron failed to run (it happens), or your script fails, PushMon will alert you by 3:30 AM. You can get alerted by email, SMS, phone, IM or Twitter, and the service is free.

Disclaimer: I am associated with PushMon.

share|improve this answer
add comment

I built a simple tool for this type of monitoring - https://cronitor.io

It allows you to set both intervals (every 24 hours) and durations (greater than 10 minutes, less than 2 minutes, etc), and then receive email/SMS alerts if your cron job (or any other automated task) doesn't run according to the rules you defined.

The tool is free for individual monitors and paid plans are available for those with multiple monitoring needs.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.