Problem I'm trying to solve: - Monitor a log file's age and if it is older than a certain threshold restart a bunch of associated services.
Solution so far: - This is fairly straightforward and we can achieve this through event handlers and the standard file age plugin.
The tricky part of this problem is that the log gets a new entry when the services are restarted. In this scenario I am restarting the service through the event handler as soon as I see failure (1st soft state i.e., not waiting for rechecks). Now, the original check was to see if the log file is older than 90 minutes and recheck interval is 15 mins. The problem is, on the second recheck the file age will be less than 90 minutes and so a notification will not be sent out.
My question is this: - Is there a way to change the parameters of the check in the middle of an event handler? Meaning can I lower the threshold of the check to say 10 minutes for the second recheck? - Is there a simpler solution than simply having the event handler script sleep for a certain amount of time and manually recheck (meaning through the bash script) the file's age?
A timeline to further clarify the problem (Assume we are monitoring file X to see if its age is greater than 90 minutes and the recheck time on failure is 15 minutes):
8:27 PM - last modification of X 10:00 PM - the check fails and trips up the event handler 10:00 PM - the services are restarted 10:00 PM - X gets updated because of the service restart 10:15 PM - X passes the recheck because the file's age is less than 90 minutes old.
Ideally, the check at 10:15 PM should check if the file is less than 10 minutes old to see if the log file is actually be updated.
Any help is appreciated!