Smart Notifications (Whom to contact for what) in Icinga

Question

I have Icinga 1.10 installed and I am using NagiosQL to configure it. I am stuck a bit on tweaking alerts and notifications.

Icinga has implied inheritance. Assume we have a production web server called prodweb01. It has two service checks running on it. One to check if Apache is working and one to check if a JVM is working. There are no alarm settings set for either checks and hence it inherits those settings from the host. The host inherits two host templates - Generic-Linux-Host (which has contacts for Sysadmins) and Generic-Web-Host (which has contacts for Developers). Is there anyway to write logic so that if Apache goes down, sysadmins get alerted and if JVM goes down, developers get alerted?

Is this even the best way of doing things? This approach will get tedious for us because we have hosts separated by environment and roles. So we are going to have to create prod/non-prod Generic-Linux-Host templates and prod/non-prod templates for each server role (eg. api, app, web, etc). We are going to have to do this so that the non-prod notifications don't go to SMS while the prod ones do because I haven't seen a way to tell Icinga to alert via SMS for prod and via email for non-prod. The only way to do this is it seems by creating two different contacts for the same person, one with SMS, one with email and applying it to the template or host based on it's environment.

Edit: I realize that someone may point out that why don't I just define the alarm/notification settings in the service itself? Yes, that can be done but that still leaves the question unanswered about how to separate alerts for prod from non-prod. In this scenario, the solution would be to create two versions of the same service check and set SMS alerts in one and email in the other.

Answer 1

Solved my problem as such.

Create two separate notification scripts in Icinga in commands.cfg
define command {
        command_name                    Custom-Host-Notify
        command_line                    /etc/icinga/Custom-Host-Notify "$HOSTNAME$" "$HOSTSTATE$" "$SHORTDATETIME$" "$HOSTOUTPUT$" "$HOSTADDRESS$" "$NOTIFICATIONTYPE$" "$HOSTDISPLAYNAME" "$NOTIFICATIONAUTHOR$" "$NOTIFICATIONCOMMENT$" "$HOSTNOTESURL$" "$CONTACTPAGER$" "$CONTACTEMAIL$"
        register                        1
}

define command {
        command_name                    Custom-Service-Notify
        command_line                    /etc/icinga/Custom-Service-Notify "$HOSTNAME$" "$HOSTALIAS$" "$SERVICEDESC$" "$SERVICESTATE$" "$SHORTDATETIME$" "$SERVICEOUTPUT$" "$HOSTADDRESS$" "$NOTIFICATIONTYPE$" "$SERVICEDISPLAYNAME$" "$NOTIFICATIONAUTHOR$" "$NOTIFICATIONNAME$" "$SERVICENOTESURL$" "$CONTACTPAGER$" "$CONTACTEMAIL$"
        register                        1
}

Define contacts.cfg as such

define contact {
        contact_name                    John Doe
        alias                           Senior Systems Engineer
        contactgroups                   SystemAdministrators
        host_notifications_enabled      1
        service_notifications_enabled   1
        host_notification_period        24x7
        service_notification_period     24x7
        host_notification_options       d,u,r,f,s
        service_notification_options    w,u,c,r,f,s
        host_notification_commands      Custom-Host-Notify
        service_notification_commands   Custom-Service-Notify
        email                           [email protected]
        ##This is how we send SMS to verizon phones using email.
        pager                           [email protected]
        register                        1
        }

In icinga.cfg, enable this -

enable_environment_macros=1

In /etc/icinga/, create the two scripts that are being called -

custom-host-notify

#!/bin/bash
    ##Icinga passes variables in this format - Custom-Host-Notify "$HOSTNAME$" "$HOSTSTATE$" "$SHORTDATETIME$" "$HOSTOUTPUT$" "$HOSTADDRESS$" "$NOTIFICATIONTYPE$" "$HOSTDISPLAYNAME" "$NOTIFICATIONAUTHOR$" "$NOTIFICATIONCOMMENT$" $HOSTNOTESURL$" "$CONTACTPAGER$" "$CONTACTEMAIL$"

    ##The quotes are essential since some of the variables have spaces and we need to capture the variable as a single variable. Bash sees space separated words as separate variables.
    #Define variables
    HOSTNAME="$1"
    HOSTSTATE="$2"
    SHORTDATETIME="$3"
    HOSTOUTPUT="$4"
    HOSTADDRESS="$5"
    NOTIFICATIONTYPE="$6"
    HOSTDISPLAYNAME="$7"
    NOTIFICATIONAUTHORS="$8"
    NOTIFICATIONCOMMENT="$9"
    HOSTNOTESURL="${10}"
    CONTACTPAGER="${11}"
    CONTACTEMAIL="${12}"
    ENV=`echo $1 | awk '{print substr($0,8,1)}'`
    ##Evaluate if the environment is Production.Please note, your logic may differ here. This is based on our naming convention.
if [ "$ENV" == "p" ];then
    ##If Production, then send SMS
    /usr/bin/printf "%b" "$NOTIFICATIONTYPE - $HOSTNAME is $HOSTSTATE $SHORTDATETIME - Info:$HOSTOUTPUT ($HOSTADDRESS$)" | /bin/mail $CONTACTPAGER$

    else
    /usr/bin/printf "%b" "$SHORTDATETIME - Info:\n\n$HOSTOUTPUT\n\n$NOTIFICATIONTYPE - Host $HOSTNAME is $HOSTSTATE\n\n(Address: $HOSTADDRESS Displayname: $HOSTDISPLAYNAME)\n\nNotes: $NOTIFICATIONAUTHOR $NOTIFICATIONCOMMENT\n$HOSTNOTESURL\n\nUrl:\nhttps://icinga-tech.apartmentsext.com/icinga/cgi-bin/extinfo.cgi?type=1&host=$HOSTNAME\n\n" | /bin/mail -s "$NOTIFICATIONTYPE - $HOSTNAME is $HOSTSTATE" $CONTACTEMAIL
    fi

custom-service-notify

#!bin/bash
##Icinga passes variables in this format - Custom-Service-Notify "$HOSTNAME$" "$HOSTALIAS$" "$SERVICEDESC$" "$SERVICESTATE$" "$SHORTDATETIME$" "$SERVICEOUTPUT$" "$HOSTADDRESS$" "$NOTIFICATIONTYPE$" "$SERVICEDISPLAYNAME" "$NOTIFICATIONAUTHOR$" "$NOTIFICATIONNAME$" $SERVICENOTESURL$" "$CONTACTPAGER$" "$CONTACTEMAIL$"

##The quotes are essential since some of the variables have spaces and we need to capture the variable as a single variable. Bash sees space separated words as separate variables.
#Define variables
HOSTNAME="$1"
HOSTALIAS="$2"
SERVICEDESC="$3"
SERVICESTATE="$4"
SHORTDATETIME="$5"
SERVICEOUTPUT="$6"
HOSTADDRESS="$7"
NOTIFICATIONTYPE="$8"
SERVICEDISPLAYNAME="$9"
NOTIFICATIONAUTHORS="${10}"
NOTIFICATIONNAME="${11}"
SERVICENOTESURL="${12}"
CONTACTPAGER="${13}"
CONTACTEMAIL="${14}"
ENV=`echo $1 | awk '{print substr($0,8,1)}'`
##Evaluate if the environment is Production.Please note, your logic may differ here. This is based on our naming convention.
if [ "$ENV" == "p" ];then


##If Production, then send SMS
echo "$NOTIFICATIONTYPE - $HOSTALIAS/$SERVICEDESC is $SERVICESTATE on $HOSTADDRESS at $SHORTDATETIME ($SERVICEOUTPUT)" | /bin/mail $CONTACTPAGER

else
##If not production, send email
/usr/bin/printf "%b" "$SHORTDATETIME - Info:\n\n$SERVICEOUTPUT\n\n$NOTIFICATIONTYPE - Service $SERVICEDESC on Host $HOSTALIAS is $SERVICESTATE\n\n(Address: $HOSTADDRESS Displayname: $SERVICEDISPLAYNAME)\n\nNotes: $NOTIFICATIONAUTHOR $NOTIFICATIONCOMMENT\n$SERVICENOTESURL\n\nUrl:\nhttps://icinga-tech.apartmentsext.com/icinga/cgi-bin/status.cgi?host=$HOSTNAME\n\n" | /bin/mail -s "$NOTIFICATIONTYPE - $HOSTALIAS/$SERVICEDESC is $SERVICESTATE" $CONTACTEMAIL
fi

You can always customize the messages and variables to your liking.

Restart Icinga and you are good to go.

Edit: I decided to forgo implied inheritance and just set the notifications in the service-templates. You can set the notifications in the services itself.