I have a probe monitors the health of my web services. It curls a url and ensures that the services return data. The curl call is set to timeout in 3 seconds. So, slow responses should fire off alarms. However, I don't want the first timeout to be considered a severe issue. Rather, I'd like the probe track from run-to-run how many timeouts it has seen and increase severity of the alarms accordingly. Obviously, this can be done a custom datastore that keeps track of this info but I'm looking for a solution that is built into the Nimsoft system. Is this possible?
There is some terminology I would like to clarify. Are you talking about events or alarms?
You can customize the event thresholds on what triggers the alarm but once an event threshold is breached, an alarm is generated, at whatever level you determine.
Within the NAS, you can configure Auto-Operator rules to escalate the severity of a previously issued alarm, at customizable intervals but this does not address your issue.
You can change the setup to sample across an interval, prior to firing the alarm and this may be what you are after.