Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm currently running a single Nagios instance. From time to time, I'm getting false alarms about timeouts - for example, it says that HTTP is down on some server, but when I open it in my browser several seconds later, it loads fast, and in general there is no trace of an error.

What can I do to reduce such false alarms?

I'm guessing that it's because of transient network issues on my monitoring server. I guess that setting up another monitoring server on a different network would greatly help, but how do I plug it into Nagios?

Is it at all possible with Nagios or do I have to switch to another monitoring system? I like my configs and, if possible, I'd like to stay with Nagios or something compatible (Icinga?)

share|improve this question
3  
You want to set up another monitoring server, which will have its own transient network issues, and send you alerts about them, in order to reduce the number of alerts you get about transient network issues? You're Doing IT Wrong my friend... –  voretaq7 Jul 10 '12 at 20:24
    
@voretaq7 I think the OP is suggesting a system where alerts are only triggered when both monitoring servers report a problem with a service. –  mgorven Jul 10 '12 at 21:03
1  
@mgorven ...but what happens if one of the monitoring servers is down? Do I send all the alerts, or none of them? Distributed monitoring is a hard problem, and Nagios is a bad solution for it. –  voretaq7 Jul 10 '12 at 21:11
    
@voretaq7 If it wasn't a hard problem, I would have written a script in 15 minutes and not bother the Server Fault community asking for a ready made solution. What do you suggest if Nagios is not the right answer? –  GDR Jul 17 '12 at 16:37
    
@GDR I suggest thinking long and hard, about all of the consequences of what you're proposing, and then deciding which cost/benefit trade-off makes the most sense in your environment. I'm not saying Nagios is the wrong solution (though I've got no love for it personally), but reconciling distributed monitoring is non-trivial, and THAT is probably not the solution you want because of the added complexity. You've received a number of excellent suggestions in the answers below... –  voretaq7 Jul 17 '12 at 17:21

3 Answers 3

Increase the threshold for alerting. For example, don't have it alarm after 1 failure. Have it alarm after 3 failures and put a sane interval (1 minute, 2 minutes) between re-checks. This means that you'll be notified if it's down for 4-5 minutes, not if you have "transient network issues" on your monitoring server.

share|improve this answer
1  
I would also add that you shouldn't assume monitoring is broken because you're getting alerts. Make sure your alert thresholds are sane (expect a dynamic page to generate in 1 second, but sometimes it takes 1.5? Maybe the response timeout should be higher - like 5-10 seconds?), and then respect the word of your monitoring system... –  voretaq7 Jul 10 '12 at 20:25
    
I'm doing it right now and it didn't help. The network issues sometimes last longer than a few minutes, and I certainly don't want half an hour delay when a server is down. –  GDR Jul 17 '12 at 16:33
1  
If you have network issues that cause connectivity problems longer than a few minutes, you don't have transient network problems, you have real network problems. You should fix that. –  MDMarra Jul 17 '12 at 16:55

Increase your thresholds for an alert. In fact you may be better off doing this kind of monitoring from a script which logs the transaction times, sends notices to Nagios, and periodically analyzes its log of recent turnaround times to send an alert only if there is a bad trend developing.

This lets you set the threshold higher so that it does not alert on EVERY transaction that takes too long, but still alerts you if a moving average transaction time gets too high. You will be a bit slower to respond to a real major problem, but you won't be worn out by so many false alarms.

In any case, real major problems that are your fault (not acts of god or the data center operator) are better dealt with via automated restarts and reboots because that is the fastest way to fix such problems if they are easily fixable. And if they are not easily fixable, a delay of a couple of minutes caused by a higher threshold will make no real difference to how you recover from the problem.

Don't be afraid to experiment with thresholds. When you are available to respond to alarms, experiment with lower thresholds and see what happens. Bump up the thresholds when you are off on a date, and do a review afterwards to see whether or not anything important was missed.

share|improve this answer
    
I agree with everything here except Bump up the thresholds when you are off on a date, and do a review afterwards to see whether or not anything important was missed. -- that's very irresponsible. Experiment and set your thresholds when you are available to monitor them, and when you have working thresholds don't touch them without good, solid reasons - ESPECIALLY if you're not able to respond instantly if something breaks. Your pager interrupting your date is better than your boss calling screaming that the website is DOWNNNNNNN!!!!!!!1111ONEELEVENTY –  voretaq7 Jul 10 '12 at 21:13
    
Basically, you need to know what is the definition of normal and that requires experimentation to find both the upper and the lower boundaries of that definition. –  Michael Dillon Jul 10 '12 at 22:03

First you have to trace the reason why the http-request times out.

If you have more than 50 servers and more than 5 monitored values per server it is probable that Nagios itself is the culprit.

It generates a request for every monitoring event and produces lots of Network interrupts.

Instead of raising the alerting threshold, you can change the timeout and retry values in the http-check-method.

share|improve this answer
    
It's a good point but not my case. The monitoring server is barely loaded and far from the numbers you cited. –  GDR Jul 17 '12 at 16:35
    
@GDR so now things get interesting. Can you do the check "by hand" by using the nagios/libexec-script "by hand"? Is it also slow? –  Nils Jul 17 '12 at 20:30
    
It's not slow. In other words, load is close to 0.05. –  GDR Jul 18 '12 at 16:01

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.