I have an environment where I centrally monitor with Nagios a number of customer environments. The idea here is not to fully manage these environments, but rather let the environment be largely autonomous and function as an escalation path for issues that can not be resolved directly.

I've found using NSCA as a notification method to have some advantages over the more common distributed monitoring system using an obsessive compulsive command. Namely, I can use the notification logic in Nagios to configure which issues get escalated and under what conditions. For example, a customer can acknowledge a problem before we get involved, something not possible with obsessive compulsive based distributed configurations.

The problem is that when a notification is lost, the two instances can lose synchronization. For failed services this is easily solved with a serviceescalation; if a failure notification is not received, another will be sent again shortly. However, it seems to me that recovery notifications are never re-sent, regardless of the escalation settings. So, if the central location receives a failure notification but misses the recovery notification, the two locations will remain out of sync forever.

This solution would be ideal if I could cause notifications of OK state to be re-sent on some interval, even if there's been no change, but I see no way to do that. Failing that, what other solutions have I not considered?

  • Best to fix the root-cause of the issue. NSCA doesn't guarantee delivery of updates. You can solve this by using NSCA over a transport agent that does guarantee message delivery, such as OpenMQ. This way, any undelivered messages would be queued and automatically replayed, when your network layer has recovered. – Jim Black Mar 23 '13 at 20:16

After substantial research, the answer seems to be simply "no". Recovery notifications are not re-sent under any circumstances.

An alternative is to use an obsessive compulsive service/host command to invoke NSCA after every check. This is a far more common configuration.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.