Nagios custom variables for object inheritance

Question

In our Nagios setup we're using templates and object inheritance for services and hosts.

#Le Hosts
define host{
    use            linux-nrpe,linux-dc3,linux-cassandra
    host_name      tigris
    alias          tigris
    address        192.168.4.72
    }

define host{
    use            linux-nrpe,linux-dc3,linux-cassandra
    host_name      euphrates
    alias          euphrates
    address        192.168.4.177
    }

#Le Templates
define host{
    name           linux-nrpe
    use            all-hosts
    hostgroups     linux-nrpe
    contact_groups rhands,usergroup1,opcomms
    register       0
}

#Le Services
define service{
    hostgroup_name      linux-nrpe
    use                 high-priority-service,graphed-service
    service_description Load
    check_command       check_by_nrpe!check_load!5,5,6!9,9,9
    contact_groups      rhands,usergroup1,opcomms
    }
[...etc...]

The problem with this setup is all servers in the linux-nrpe group trigger alerts when their load levels hit whatever is defined in the service, but our workhorse servers might run 24/7 at a load of 20 but our DB servers sit quite happily at ~1 unless something goes wrong, so we find the system sending out too many alerts or having to ignore/not alert on things. Defining individual service definitions for each server (lots of them) would take ages, what we'd really like to do is something like

define host{
    name           linux-nrpe
    use            all-hosts
    hostgroups     linux-nrpe
    contact_groups rhands,usergroup1,opcomms
    register       0
    perf_load      2,2,3 5,5,6
    perf_mem       95% 97%
    [...more...]
    }

define service{
    hostgroup_name      linux-nrpe
    use                 high-priority-service,graphed-service
    service_description    Load
    check_command       check_by_nrpe!check_load!$perf_mem$
    contact_groups      rhands,usergroup1,opcomms
    }

I looked through the docs and couldn't see anything, unless I'm missing something. Any ideas?

Answer 1

We have a quite similar solution running here in our Nagios Monitoring. Custom Host/Service Variables have to start with an underscore on definition and on reference you have to add _HOST or _SERVICE as prefix and all uppercase as name.

Therefore you perf_load and perf_mem custom variable has to be defined as

define host {
    [..]
    _perf_load      2,2,3 5,5,6
    _perf_mem       95% 97%
    [..]
}

and referenced as

define service {
    [..]
    check_command   check_by_nrpe!check_load!$_HOSTPERF_LOAD$
    [..]
}

A snippet from a running config of our Nagios:

define host {
       host_name               target
       alias                   target
       address                 target
       use                     tmpl_host
       _gprs_address           192.168.0.1
}
[...]
define service {
        host_name               target
        service_description     GPRS ping
        use                     tmpl_service_ping
        check_command           check_fping-by-ssh!-H 1.2.3.4 -S $_HOSTGPRS_ADDRESS$ -n 7 -t 1000 -w 1000 -c 2000
        event_handler           check_restart-GPRS-PPP
        notes_url               https://wiki.
        contact_groups          admin_allday
}

You find more details in the Nagios Documentation.

For the reference, this work also fine in Icinga.

Answer 2

You can also define the thresholds in the NRPE config, on the hosts themselves. This isn't practical if you have more than a few dozens hosts, unless you have some sort of conf management (something like puppet, or even just git/hg/svn/whatever) and use 'includes' in nrpe.cfg.

Lairsdragon's suggestion is much better, though. The one thing I would add is:

It can be helpful to name custom object vars with two leading underscores ($__FOO), so they can be called as "$_HOST_FOO".