4

I'm not sure how to explain this. Once one starts having dozens or hundres of servers, each running some sort of periodic jobs (mostly from cron), there is a problem of making sure (or as sure as possible) that these tasks are actually ran. I mean, I get an email if a job fails fails, and no mail if it succeeds, but also no mail if it doesn't run for whatever reason. Sure, I could change them and have them send a "successfully ran" email, only to be flooded by mails that most of the time I don't want to see.

Basically, I want to be notified only if:

  • a task ran and failed
  • a task didn't run at the expected time.

Is there a way to do this?

5
  • What do you mean with "doesn't run"? For what reason? If I misspell a command or cron doesn't find the command, I get an email. Nov 16, 2012 at 10:03
  • I have seen your answer only in my notification area. It seems to be lost or deleted. Nov 16, 2012 at 10:18
  • One reason could be because someone made a change to the crontabs without telling me. Another reason could be that the script it runs was replaced by an empty file, or by a symlink to another command. Or the cron daemon crashed. Or if the task is to enqueue some job into some batch system, perhaps the batch system scheduler is not executing the jobs because the first job in then queue is blocked, and new jobs are stuck in the queue. Or an rsync job isn't doing anything because the source of its data isn't getting updated. Etcetera, etcetera.
    – persson
    Nov 16, 2012 at 10:21
  • What I probably want is something that observes the expected effects that my tasks should produce, and warns if it doesn't see them. But, without having to write a whole set of dedicated scripts only to do that, or even using some monitoring system; both things seems overkill to me.
    – persson
    Nov 16, 2012 at 10:24
  • What you describe is some sort of monitoring system. Have you already seen @Andrey's answer? I think this is a good start. Combined with logcheck or a similar tool this could accomplish, what you need. Nov 16, 2012 at 10:29

2 Answers 2

4

I assume your are running UNIX/LINUX system if you are using cron, so the most simple way to monitor "a task didn't run at the expected time" is remote logging to one(?) log server, something like this in your syslog.conf:

cron.!=info;   @logserver.yourdomain.tld

Then you can browse those logs for errors or just add some log parsing script to do this job.

But anyway there always will be possibility that logging is broken and you just do not receive error messages.

2
  • +1 for a smart solution Nov 16, 2012 at 10:24
  • Yes, see above, I used a bad wording (sorry about that). There are tasks that may run, but don't produce the expected results, so I'd have evidence of their running in the syslog, but I wouldn't know whether they did what they were expected to do.
    – persson
    Nov 16, 2012 at 10:27
4

I don't think there is something already done about that. You could set all tasks to log in a single file their status: timestamp, server, task, status, other info.

And than you parse this file periodically and send email when status is failed or warning or tasks have old timestamps. In order to be sure that this script was successful it will send an email even if everything is ok.

This way you won't be flooded by emails, it will be just one email from time to time.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .