I have an environment where I centrally monitor with Nagios a number of customer environments. The idea here is not to fully manage these environments, but rather let the environment be largely autonomous and function as an escalation path for issues that can not be resolved directly.
I've found using NSCA as a notification method to have some advantages over the more common distributed monitoring system using an obsessive compulsive command. Namely, I can use the notification logic in Nagios to configure which issues get escalated and under what conditions. For example, a customer can acknowledge a problem before we get involved, something not possible with obsessive compulsive based distributed configurations.
The problem is that when a notification is lost, the two instances can lose synchronization. For failed services this is easily solved with a serviceescalation; if a failure notification is not received, another will be sent again shortly. However, it seems to me that recovery notifications are never re-sent, regardless of the escalation settings. So, if the central location receives a failure notification but misses the recovery notification, the two locations will remain out of sync forever.
This solution would be ideal if I could cause notifications of OK state to be re-sent on some interval, even if there's been no change, but I see no way to do that. Failing that, what other solutions have I not considered?
