2

I have been a nagios user for a long time.

Of late, as the size of our server fleet grew, so did the number of alerts from nagios. Signal-to-noise ratio has become very low. eg. When a common service fails - all my load-balanced web-server which use that service & hence check for it start alerting. That mixed with system alerts possible from that service appearing in different order lead to a lot of noise.

I can spend a lot of time & ensure my nagios configurations are good, but it is increasingly becoming unmanageable. I am looking for a tool (or nagios plugin) that does de-duplication & intelligent suppression of alerts. Also, I would want "issues"/outages to be tracked in a trouble ticketing system - so that there is 1 place for anyone to get a good handle of whats happening with an issue. And also look at the archive.

Yes, I can do it to some extent in Nagios - but its not great.

While looking I found tons of tools ( http://www.slac.stanford.edu/xorg/nmtf/nmtf-tools.html#public ) but nobody seems to be talking of issues like de-duplication, issue tracking & management

2 Answers 2

1

I'd say your best bet is OpenNMS with RT or OTRS integration. Unlike Nagios, it's a complete SNMP management solution with an FCAPS (fault/configuration/accounting/performance/security management) focus. How well it tackles each one of those categories is sort of up to the implementer. It's a great solution for people who are looking to "upgrade" from Nagios and have a Cacti server sitting around doing similar things. The integration of the performance and fault data is absolutely indispensable. The documentation is sort of behind the current state of the product, but I've been personally working on this as of late.

If you want to give it a try, go ahead and follow the quick start instructions on the opennms.org wiki, but stop at "discovery", and take a look at the new provisiond feature whitepaper. It's a great migration tool as well.

The event based system it provides triggers alarms for an alarm panel and notifications for... notifications. These can be phone calls via asterisk, pages, email, twitter, etc. When you or on-call staff are notified, you can reply to the email with the work "ack" and have the notification acknowledged and your ticket updated with start times, etc.

The separation of notifications and alarms is a great feature for your de-duplication request. Depending on what's going on, you can reduce these alarms by a reduction key and only be notified on the threshold (but still have all the alarm triggered so you have the data). There's some advanced correlation features, but I haven't really dug into it.

2

I think the first thing I would suggest is looking at dependencies. When your common service fails, the web servers that use that common service could be set as dependencies so that you would only get the alert for the common service. If one of the web servers fails and the common service is still responding, you would get the web server down message.

Typically you set up Nagios so that all machines behind a switch or on a particular circuit breaker have a dependency. When that switch fails, there is no need to alert you that all 38 machines in that rack are also down.

If you experience flap, you can alter the escalation slightly so that your service metrics are still maintained, but, you're not paged as frequently when a service flaps. Nagios's flap detection doesn't do a superb job, but, it doesn't do too badly.

As for ticketing, you can write an event notification script that imports into a trouble ticketing system automatically as we've done (clients get some exported data for their records), but, I've not researched integration into existing systems.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .