When monitoring the healthy of a server, some faults or warnings are immediately urgent but others only matter if they persist. I'm thinking of things like:
- Some software needs to be updated
- Time offset differs from NTP
If unaddressed these could become real problems, but there are already background services in place to take care of them - unattended upgrades, an NTP client service etc. There's always a short delay between the problem arising and these background processes kicking in to address them, and our monitor is sending out a series of emails in that gap - then again a minute later when the issue is fixed. I generally wake up to a large pile of "PROBLEM" emails, each with a corresponding "RESOLUTION" emails sent a minute later. The danger is that in dismissing a hundred irrelevant warnings, I could miss the one that's real.
So is there any way of instructing Icinga or Nagios to only report an issue if it's continued for more than a certain time, say 5 minutes?