0

In many projects my team faced problems with 'silent fails' of some important components. There are lot of tasks executed behind the scenes and if somethings fails (either by errors in logic or hardware problems) in most cases responsible person is not notified (or not notified instantly).

I know about heavy-weight monitoring tools that could solve some of that problems but there over-complicated and too expensive for our team.

I am interested what are your solutions for such problems.

Thanks for your responses so far. To be more accurate I am looking for something that meets following criterias:

  1. reliability - I think that relaying on solutions like cron's MAILTO or executing notification scripts if job script returns some value are not fully reliable (e.g. there are general problems with server). The fully reliable solutions are deployed on separated environment.

  2. possibility to give immediate alert to interested person (emails cannot be treated as immediate in some cases, SMS would be far better). It would be great to prevent from 'emails avalanche' when you receive email every minute with same information.

  3. requires as little knowledge about set up and configuration as possible.

  4. ability to monitor and alert when script execution exceeds some time

  5. alerting rules are maintained from one place.

I did some research and couldn't find anything that covers those criterias. Nagios (or similar tools) are close to be good enough but in my opinion they are to complicated, not user-friendly, require complicated integration. It also reuires to hire someone who is familiar with such tool or spend a lot of time to master them.

The main reason I'm asking about such solutions is that we develop in our software company solution based on interesting approach that can fullful such requirements(or most of them) and already it works pretty good in our projects. Now we are aiming to release it for community and we are looking for some solutions that can do nearly the same to make analysis of advantages and disadvantages of our approach and choose direction of development. Comments about your problems with exisiting solutions and things you really appreciate are kindly welcome as well.

Improve this question

5 Answers 5

Reset to default
1

Nagios with passive checks and then wrap your scheduled jobs to send a message (send_nsca)to your nagios server indicating what happened when they complete. If the job errors then nagios will alert.

More relevant to the problem you're seeing is that you can also set nagios to alert if it hasn't heard from your cron job for too long, so you can spot jobs that are failing silently.

All free and fairly trivial to set up.

Improve this answer
1
  • Thanks for your comment. I have to give nagios a try one day. But I think that in most cases for me and many developer installing, deplying and managinf nagios is to much effort for simple monitorring tasks (especially for people not experienced with nagios).
    – dzida
    Jun 13, 2010 at 19:59
0

You could change the command entry in crontab to be somethign like

/usr/local/bin/critical_job || /usr/local/bin/notify "critical job failed"

Then when "citical_job" exits with non-zero, "notify" would be run. Preferably, I'd do my best to catch it the the job itself and handle it there.

Improve this answer
1
  • nice, but this will not work in some cases (what if critical_job fails in some unexpected way). There is also problem of serwer downtime that is not solved by this solution. I think that monitoring and alerting should be set up on separated environment to do it with maximum relialability.
    – dzida
    Jun 13, 2010 at 20:08
0

If you don't feel that a larger-scale monitoring solution is the right match for your situation, you might consider having the administrator e-mail forwarded off of the server to someone's actual e-mail account. You can do this by adding a line such as the following to /etc/aliases:

 root: [email protected]

Where [email protected] is somebody's actual e-mail address.

Mind that your server will need a functioning MTA to send e-mail, and if there are any conditions that prevent the system from sending e-mail (ie loss of network / no space on /var), nobody will be notified.

Improve this answer
0

check out https://cronitor.io -- you can monitor a single cron job for free. When your jobs run, they ping Cronitor and if Cronitor doesn't get pinged at the right times it will send you an alert.

Improve this answer
0

Hosted Dead Man's Switch style services would be a perfect fit here. The idea in a nutshell: you set up your cron job to make a HTTP request to the hosted service just before the cron job finishes. The service sends you an alert whenever it doesn't hear from the cron job on time.

  • reliability: the hosted service runs on a different infrastructure than your cron job. For truly mission critical stuff you could ping multiple services.
  • SMS notifications is a common feature
  • minimal setup or maintenance, like with any SaaS
  • you can typically configure how "late" the cron jobs are allowed to be before you get an alert
  • the hosted service will provide a dashboard with all your registered cron jobs, their schedules, current status, next expect run time etc.

A few well known options:

(Full disclosure – I'm the founder of Healthchecks.io.)

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .