Nagios (or a similar tool) with more variety in alert levels?

Question

I'm looking at monitoring and alerting for some of our business metrics in a web app (pageviews, signups, etc). We already use Nagios and Munin for a wide variety of server monitoring and alerting cases, which is why I've started there.

I could write custom plugins for Nagios that calculate our statistics / control charts and check for when these metrics dip below desirable levels (Warning and Critical), but I'd also like to know when these metrics spike above the expected levels (lots more signups - we did something right!).

Is there a way to create custom alert levels in Nagios or Munin to accomodate these positive alerts, or is there another tool I should be looking at to solve for this case? The ideal tool would:

1. Include more alert levels (Critical, Warning, OK, Improved, Spiking)
2. Allow me to see additional data about the report that generated the alert (expected value of the metric and observed value)
3. (Nice to have) allow me to graph the history of the metric so I can visualize the observations after receiving the alert

Answer 1

Many NAGIOS plugins already do exactly what you want, and allow WARNING and CRITICAL thresholds to be specified as a range, alerting when the value is outside (or, optionally, inside) that range. For example, from /usr/lib/nagios/plugins/check_procs -h:

[...]
Usage:
check_procs -w <range> -c <range> [-m metric] [-s state] [-p ppid]
 [-u user] [-r rss] [-z vsz] [-P %cpu] [-a argument-array]
 [-C command] [-t timeout] [-v]
[...]
 -w, --warning=RANGE
   Generate warning state if metric is outside this range
 -c, --critical=RANGE
   Generate critical state if metric is outside this range
[...]
RANGEs are specified 'min:max' or 'min:' or ':max' (or 'max'). If
specified 'max:min', a warning status will be generated if the
count is inside the specified range
[...]

So this plugin already has the idea that "X is problematic if it's more than C or less than B, and it's an impending disaster if it's less than A or more than D" (for A<B<C<D). That sounds to me very much like what you asked for, and no rearchitecting of NAGIOS (into which the OK/WARN/CRIT concept is pretty heavily embedded) is needed.

Answer 2

I use nagios for alerting and ganglia for metrics gathering. Often times my tool that sends data into ganglia (via gmetric) also sends alerts into nagios (via send_nsca, aka passive alerts.) If you write your tool properly it can detect changes from the last period to the current period. This way you can, for example, detect changes in certain metrics. Is the average response time at this time X% higher/lower than the last time? Did the queue size increase/decrease X% more/less?

I used munin for a while, but it didn't have the granularity of ganglia. Also, creating a new metric in ganglia is as easy as calling gmetric. BAM, you have a new graph. I also find it useful to have the last deploy time displayed on our metrics graphs (just a vertical line showing the time of the deploy.) That way you can better visualize what's going on and when.