Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I can view "Currently failing" nodes in the puppet dashboard web GUI, and even drill into it to see what policy has gone awry. I can also see if nodes are not reporting.

Is there any way I could get these failures as email alerts as well?

share|improve this question
    
Haven't played much with Puppet yet, but is there a command-line equivalent to the dashboard that could be fed to a Nagios plugin (or something similar)? –  voretaq7 Nov 29 '11 at 17:03

5 Answers 5

up vote 2 down vote accepted

Puppet's "tagmail" report might do what you want. See http://docs.puppetlabs.com/references/0.25.5/report.html#tagmail for more information.

share|improve this answer
    
The tagmail.conf supports err: foo@foo.com which seems to do what I am looking for. –  Kyle Brandt Dec 1 '11 at 2:31
    
Awesome, glad I could help! –  Paul Lathrop Dec 1 '11 at 18:32

Foreman Email is possibly the best way to get such reports. You could run it along side Dashboard.

share|improve this answer

I had the same issue, and rather than going through the effort of installing Foreman, I ended up writing a quick and dirty shell/cron script to curl and grep puppet dashboard, and email the results. Thought it might be helpful to post. Surely can be cleaed up but works for me at the moment.

#!/bin/bash

UNRESPURL="http://puppet/nodes/unresponsive"
ERRORURL="http://puppet/nodes/failed"
EMAIL=my@email.address.com

NUMUNRESPNODES=`curl ${UNRESPURL} 2> /dev/null | grep -Po '(?<=<a href="/nodes/unresponsive">)([^</a>]*)' | grep -v "Unresponsive"`
NUMERRORNODES=`curl ${ERRORURL} 2> /dev/null | grep -Po '(?<=<a href="/nodes/failed">)([^</\a>]*)' | grep -v "Failed"`

# Check if there are any unresponsive nodes, before emailing
if [ ${NUMUNRESPNODES} -gt 0 ]; then
        # Email an alert
        echo "There are ${NUMUNRESPNODES} puppet nodes that haven't reported back to the puppetmaster. Please check ${UNRESPURL} for more info." | mail -s "ALERT: ${NUMUNRESPNODES} puppet nodes are unresponsive" ${EMAIL}
fi

# Check if there are any nodes with errors, before emailing
if [ ${NUMERRORNODES} -gt 0 ]; then
    # Email an alert
    echo "There are ${NUMERRORNODES} puppet nodes that are reporting errors. Please check ${ERRORURL} for more info." | mail -s "ALERT: ${NUMERRORNODES} puppet nodes are reporting errors" ${EMAIL}
fi

exit 0
share|improve this answer
    
There are errors on -gt 0 on both lines. How does the script gain access to the dashboard? –  nicoX Jan 30 at 17:35

What we currently do at Stack Exchange (Saw some upvotes on this question so figured I would add a updated answer):

Our monitoring agent scollector reads the run summary file in /var/lib/puppet/state/last_run_summary.yaml. When converted to a struct the relevant porition of that file looks as follows:

Resources struct {
    Changed         float64 `yaml:"changed"`
    Failed          float64 `yaml:"failed"`
    FailedToRestart float64 `yaml:"failed_to_restart"`
    OutOfSync       float64 `yaml:"out_of_sync"`
    Restarted       float64 `yaml:"restarted"`
    Scheduled       float64 `yaml:"scheduled"`
    Skipped         float64 `yaml:"skipped"`
    Total           float64 `yaml:"total"`
} `yaml:"resources"`

The agent then sends the "Failed" resource count ) which acts as a gauge. I can then have a rule in our monitoring system bosun that alerts if there have been failed resources consistently for a failed period of time:

$t = 2h
$notes = This alert will trigger when a host has consistently had failed resources for longer than $t
$q = min(q("sum:1h-min:puppet.run.resources{resource=failed,host=*}", "$t", ""))
warn = $q

This method of getting the stats from last_run_summary.yaml should be applicable to any monitoring system really.

share|improve this answer

I'm looking to get the Log>Level>err message, which as well can create an err message on a successful run. This is generated by the log file, and can be seen in the dashboard:

err Failed to call refresh: Could not restart Service[haproxy]: Execution of '/sbin/service haproxy restart' returned 1:

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.