0

Problem I'm trying to solve: - Monitor a log file's age and if it is older than a certain threshold restart a bunch of associated services.

Solution so far: - This is fairly straightforward and we can achieve this through event handlers and the standard file age plugin.

The tricky part of this problem is that the log gets a new entry when the services are restarted. In this scenario I am restarting the service through the event handler as soon as I see failure (1st soft state i.e., not waiting for rechecks). Now, the original check was to see if the log file is older than 90 minutes and recheck interval is 15 mins. The problem is, on the second recheck the file age will be less than 90 minutes and so a notification will not be sent out.

My question is this: - Is there a way to change the parameters of the check in the middle of an event handler? Meaning can I lower the threshold of the check to say 10 minutes for the second recheck? - Is there a simpler solution than simply having the event handler script sleep for a certain amount of time and manually recheck (meaning through the bash script) the file's age?

A timeline to further clarify the problem (Assume we are monitoring file X to see if its age is greater than 90 minutes and the recheck time on failure is 15 minutes):

8:27 PM - last modification of X 10:00 PM - the check fails and trips up the event handler 10:00 PM - the services are restarted 10:00 PM - X gets updated because of the service restart 10:15 PM - X passes the recheck because the file's age is less than 90 minutes old.

Ideally, the check at 10:15 PM should check if the file is less than 10 minutes old to see if the log file is actually be updated.

Any help is appreciated!

  • Escalations can be used to change the check interval after the initial alert, what you're asking is for it to change the actual check. I'm not sure if that can be done. – Fred the Magic Wonder Dog May 7 '15 at 22:49
0

Yes, you can do this with external commands, as documented in the Adaptive Monitoring docs.

You have the event handler submit an external command to modify the check command and/or its arguments. Be sure to change it back eventually (perhaps when the notification is sent, for example).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.