nagios (Last Notification: N/A (notification 0))

Question

I can't figure out but for some reason Nagios doesn't send me an alert on some of the services that I monitor, I was wondering if someone can help me out a bit.

this is service

define service {
    hostgroup_name          nrpe-disk
    service_description     disk
    use             alarm,check
    check_command           check_nrpe1!check_disk
    register            1
    }

here is my command

define command {
       command_name                         check_nrpe1
       command_line                         $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

here is service template

define service {
       name                                 alarm
       notification_interval                0
       notification_period                  247
       notification_options                 w,u,r,c,f,s
       notifications_enabled                1
       contact_groups                       staff
       register                         0

}

define command {
       command_name                         notify-host-by-email
       command_line                         /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$
}   

define command {
       command_name                         notify-host-by-pager
       command_line                         /usr/bin/printf "%b" "Notification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$" | /usr/bin/mail $CONTACTPAGER$
}   

define command {
       command_name                         notify-service-by-email
       command_line                         /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$" | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$
}   

define command {
       command_name                         notify-service-by-pager
       command_line                         /usr/bin/printf "%b" "Notification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nAdditional Info:\n\n$SERVICEOUTPUT$" | /usr/bin/mail $CONTACTPAGER$
}   

define contactgroup {
    contactgroup_name                   staff
    alias                               XXXXXXXX
    members                             XXXXXXXX XXXXXXXX
    }   

define contact {
    contact_name                        XXXXXXX XXXXXX
    alias                               XXXXXXXXX
    host_notifications_enabled          1
    service_notifications_enabled       1
    host_notification_period            247
    service_notification_period         247
    host_notification_options           d,u,r,f,s
    service_notification_options        w,u,c,r,f,s
    host_notification_commands          notify-host-by-email,notify-host-by-pager
    service_notification_commands       notify-service-by-email,notify-service-by-pager
    email                               XXXXXXXXX@XXXXXXXX.XXX
    pager                               XXXXXXXXXX@messaging.sprintpcs.com
    }

Answer 1

I don't see it mentioned in your question or any of the answers but have you checked the nagios.cfg file for the enable_notifications setting? Is it set to 0 or 1?

Also is this only happening on a given service notification or is it affecting all notifications? If it's all I would suspect that enable_notifications is set to 0 in the nagios.cfg file. If so try changing it to 1 and restarting Nagios.

Answer 2

The nagios debug log helped me diagnose a similar issue. It was near the end of my nagios.cfg

debug_level=32
debug_file=/var/log/nagios3/nagios.debug
debug_verbosity=2

restart nagios, then tail the nagios.debug file; it'll show you something like:

[1416254305.106042] [032.0] [pid=928] ** Service Notification Attempt ** Host: 'myhost', Service: 'myservice', Type: 0, Options: 0, Current State: 2, Last Notification: Wed Dec 31 19:00:00 1969
[1416254305.106106] [032.0] [pid=928] Notification viability test passed.
[1416254305.106114] [032.1] [pid=928] Current notification number: 1 (incremented)
[1416254305.106120] [032.2] [pid=928] Creating list of contacts to be notified.
[1416254305.106137] [032.1] [pid=928] Service notification will NOT be escalated.
[1416254305.106174] [032.1] [pid=928] Adding normal contacts for service to notification list.
[1416254305.106185] [032.2] [pid=928] Adding contact 'mycontact' to notification list.
[1416254305.106219] [032.2] [pid=928] ** Attempting to notifying contact 'mycontact'...
[1416254305.106449] [032.2] [pid=928] ** Checking service notification viability for contact 'mycontact'...
[1416254305.106463] [032.2] [pid=928] We shouldn't notify this contact about CRITICAL service states.
[1416254305.106474] [032.0] [pid=928] No contacts were notified.  Next possible notification time: Wed Dec 31 19:00:00 1969
[1416254305.106481] [032.0] [pid=928] 0 contacts were notified.

In my case, I had a bad contact service_notification_options, which prevented the initial notification from going out, which blocked all my escalations (if we never send notification 0, we can't ever get to notification 1)

Answer 3

My first step would be to try defining all the parameters directly rather than depending on inheritance. Avoid groups, too, if you can. This at least gives you a clean starting place to debug from.

define service {
    host_name                  XXXXXXX
    service_description        disk
    notification_interval      0
    notification_period        247
    notifications_enabled      1
    contacts                   XXXXXXX
    check_command              check_nrpe1!check_disk
    }

You might also want to try switching notification_interval to something else, even leave it out (defaults to 60). Just until you get it working.

Answer 4

Do you have a "notify-by-email" section in your commands config file? Here's the relevant section from mine:

'notify-by-email' command definition

define command{ command_name send-email command_line /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$" | /bin/mail -s "** $NOTIFICATIONTYPE$ alert - $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$ }

Can your system send mail in general? Can you use "mail -s test foo@bar.com", then type some text, a blank line and finally Ctrl-D to send?

Answer 5

changing

register 0

to

register 1

in your "alarm" service might help.

Answer 6

Where's the service-template for check ? Since you're using with templates alarm and check. I guess check overwrites alarm...

define service {
    hostgroup_name          nrpe-disk
    service_description     disk
    use             alarm,check
    check_command           check_nrpe1!check_disk
    register            1
    }

perhaps notification_failure_criteria can help you

Do you really have a notification_period called 247 or rather 24x7?

You get a correct feedback from your command (so there should be a alarm?)? you tested the command on the command-line (as your nagios-user)?