Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Nagios in its standard usage monitors with point-in-time checks: either something is - or is not - true.

Other tools like SGI's PCP, HP's MeasureWare, and SEC provide monitoring over time - monitoring things like average disk access time over the last five minutes, or other similar items. Is there anything like this for Nagios? I'm already running NDOUtils, which seems like a natural source for such data.

I'd like to have something that would monitor and fire off alarms based on a time-based check using historical data. Is there anything like this for Nagios?

share|improve this question
    
I've still not found an add-on which basically allows you to interface with data stored by NDOUtils for time-based alarms. However, with a combination of a standard plugin plus SEC running against nagios.log I should be able to do what I want. If only there were an add-on... –  Mei Feb 10 '11 at 3:01

5 Answers 5

I wrote a Nagios check plugin using historical data from sar that you might be interested in. Even if it's not useful to you out of the box, you can easily use it as a starting point for more sophisticated checks.

share|improve this answer
    
Good idea - though it only works for system data contained in the sar reports. I'm specifically interested in working with custom data. –  Mei Feb 9 '11 at 23:11

You say

Usage example: Generate Alarm if CPU Load is over 95% for 10 minutes.

but NAGIOS already does this. If, for example, you don't want to know about a problem until it's been that way for thirty minutes, try (in the service definition)

max_check_attempts              6
retry_interval                  5

This will cause the service to be checked at five-minute intervals even after it goes SOFT ERROR, but not to go hard - and notify - until the sixth successive soft error (6 * 5mins = 30 mins).

If that's not what you wanted, can you explain how it falls short?

Edit: you note that this works for you, but that it can't cope with more complex judgement issues (eg, foo is at over 80% for more than 60% of the preceding 30 minutes).

That's true, but in my experience of deploying NAGIOS, and I've done quite a lot of it, there are very few circumstances where people really need to know something like that. They may want to know it, but when pressed, they generally don't have an engineering need. In those cases, "no" is the right answer; the monitoring tool is business-critical, and overloading it with loads of stupid I-want-a-pony tests to make some VP of Executive Laundry happy is the Wrong Thing.

On the odd occasions where they really do need something baroque, it's much better to put it in the plugin. For example, I did some work for a client to let them know how old the snapshots on any given NetApp volume were, and that was fine. Then a legitimate engineering requirement arose to check that the oldest member of the set of youngest snapshots on a given set of volumes was younger than a given limit (you may want to read that a couple of times!). I could probably have tortured NAGIOS into evaluating criteria based on the results of the multiple "snapshot age" plugins, but it was much more sensible in the long run to write a plugin that tracked and evaluated this one complex criterion on its own.

So I'd say to you: be careful that there is a good engineering need for evaluating odd criteria. In the few cases where there is, write your own plugin to track it.

share|improve this answer
    
Interesting! This may do what I want - although it is only good for some status that stays at a particular level for a particular length of time. For more sophisticated time-based alarms, this won't work: e.g., send me an alarm if this metric is over 95% for 10 out of the last 15 minutes... –  Mei Feb 9 '11 at 21:48

Have you considered using opsview or groundworks?

share|improve this answer
    
Yes... and No. The Nagios infrastructure is already in place, and these other products are more of an unknown. It would require more time getting up to speed, albeit GroundWorks has always looked interesting... –  Mei Feb 9 '11 at 21:27

Yes. The thing is called http://www.pnp4nagios.org/

It allows you to collect use the "performance data" as it is called in Nagios to graph this data with RRD.

With the Nagios way, Icinga can be interesting (it is a Nagios fork).

Another interesting thingy is http://collectd.org it is no way related to nagios, but you can define tresholds and check that condition from Nagios.

UPDATE

For the CPU is over 95% you can make a check that checks for the cpu load and do a check every minute for ten times.

share|improve this answer
    
No, that won't work - unless you can show me how PNP4Nagios can generate alarms. I already have graphs; I want alarms. –  Mei Feb 9 '11 at 21:15
    
@David: well, what kind of tool do you use for graphs then? –  cstamas Feb 9 '11 at 21:19
    
I'm already using PNP4Nagios. What I want is alarms on time based data. Usage example: Generate Alarm if CPU Load is over 95% for 10 minutes. Collectd and PNP4Nagios don't do this as far as I know. –  Mei Feb 9 '11 at 21:19
    
The problem is: using historical data requires knowledge of what has happened previously - which means referring to previously generated data. A plugin like this requires a source of historical data - which suggests NDOUtils would be part of it. –  Mei Feb 9 '11 at 21:28
    
There's plenty of other sources of historical data, though. sar has shipped with Solaris and Linux systems for ages upon ages. Plus, if you wanted, it wouldn't be difficult to write a plugin that would alarm on the values from the RRDs that PNP4Nagios maintains. –  jgoldschrafe Feb 9 '11 at 22:43

Its been years since I looked into it, but Cacti has a plugin for alerting on thresholds called like "thold" or something.

share|improve this answer
    
This question is directly related to replacing said add-on; when using things like PNP4Nagios Cacti becomes redundant - and the program's interface is only a web front-end to RRDTool. PNP4Nagios is much simpler. –  Mei Feb 10 '11 at 2:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.