In many projects my team faced problems with 'silent fails' of some important components. There are lot of tasks executed behind the scenes and if somethings fails (either by errors in logic or hardware problems) in most cases responsible person is not notified (or not notified instantly).

I know about heavy-weight monitoring tools that could solve some of that problems but there over-complicated and too expensive for our team.

I am interested what are your solutions for such problems.

Thanks for your responses so far. To be more accurate I am looking for something that meets following criterias:

  1. reliability - I think that relaying on solutions like cron's MAILTO or executing notification scripts if job script returns some value are not fully reliable (e.g. there are general problems with server). The fully reliable solutions are deployed on separated environment.

  2. possibility to give immediate alert to interested person (emails cannot be treated as immediate in some cases, SMS would be far better). It would be great to prevent from 'emails avalanche' when you receive email every minute with same information.

  3. requires as little knowledge about set up and configuration as possible.

  4. ability to monitor and alert when script execution exceeds some time

  5. alerting rules are maintained from one place.

I did some research and couldn't find anything that covers those criterias. Nagios (or similar tools) are close to be good enough but in my opinion they are to complicated, not user-friendly, require complicated integration. It also reuires to hire someone who is familiar with such tool or spend a lot of time to master them.

The main reason I'm asking about such solutions is that we develop in our software company solution based on interesting approach that can fullful such requirements(or most of them) and already it works pretty good in our projects. Now we are aiming to release it for community and we are looking for some solutions that can do nearly the same to make analysis of advantages and disadvantages of our approach and choose direction of development. Comments about your problems with exisiting solutions and things you really appreciate are kindly welcome as well.


Nagios with passive checks and then wrap your scheduled jobs to send a message (send_nsca)to your nagios server indicating what happened when they complete. If the job errors then nagios will alert.

More relevant to the problem you're seeing is that you can also set nagios to alert if it hasn't heard from your cron job for too long, so you can spot jobs that are failing silently.

All free and fairly trivial to set up.

  • Thanks for your comment. I have to give nagios a try one day. But I think that in most cases for me and many developer installing, deplying and managinf nagios is to much effort for simple monitorring tasks (especially for people not experienced with nagios). – dzida Jun 13 '10 at 19:59

You could change the command entry in crontab to be somethign like

/usr/local/bin/critical_job || /usr/local/bin/notify "critical job failed"

Then when "citical_job" exits with non-zero, "notify" would be run. Preferably, I'd do my best to catch it the the job itself and handle it there.

  • nice, but this will not work in some cases (what if critical_job fails in some unexpected way). There is also problem of serwer downtime that is not solved by this solution. I think that monitoring and alerting should be set up on separated environment to do it with maximum relialability. – dzida Jun 13 '10 at 20:08

If you don't feel that a larger-scale monitoring solution is the right match for your situation, you might consider having the administrator e-mail forwarded off of the server to someone's actual e-mail account. You can do this by adding a line such as the following to /etc/aliases:

 root: user@example.com

Where user@example.com is somebody's actual e-mail address.

Mind that your server will need a functioning MTA to send e-mail, and if there are any conditions that prevent the system from sending e-mail (ie loss of network / no space on /var), nobody will be notified.


check out https://cronitor.io -- you can monitor a single cron job for free. When your jobs run, they ping Cronitor and if Cronitor doesn't get pinged at the right times it will send you an alert.


Hosted Dead Man's Switch style services would be a perfect fit here. The idea in a nutshell: you set up your cron job to make a HTTP request to the hosted service just before the cron job finishes. The service sends you an alert whenever it doesn't hear from the cron job on time.

  • reliability: the hosted service runs on a different infrastructure than your cron job. For truly mission critical stuff you could ping multiple services.
  • SMS notifications is a common feature
  • minimal setup or maintenance, like with any SaaS
  • you can typically configure how "late" the cron jobs are allowed to be before you get an alert
  • the hosted service will provide a dashboard with all your registered cron jobs, their schedules, current status, next expect run time etc.

A few well known options:

(Full disclosure – I'm the founder of Healthchecks.io.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.