4

I can view "Currently failing" nodes in the puppet dashboard web GUI, and even drill into it to see what policy has gone awry. I can also see if nodes are not reporting.

Is there any way I could get these failures as email alerts as well?

1
  • Haven't played much with Puppet yet, but is there a command-line equivalent to the dashboard that could be fed to a Nagios plugin (or something similar)?
    – voretaq7
    Nov 29, 2011 at 17:03

5 Answers 5

2

Puppet's "tagmail" report might do what you want. See the official tagmail documentation for more information.

2
0

Foreman Email is possibly the best way to get such reports. You could run it along side Dashboard.

0

I had the same issue, and rather than going through the effort of installing Foreman, I ended up writing a quick and dirty shell/cron script to curl and grep puppet dashboard, and email the results. Thought it might be helpful to post. Surely can be cleaed up but works for me at the moment.

#!/bin/bash

UNRESPURL="http://puppet/nodes/unresponsive"
ERRORURL="http://puppet/nodes/failed"
[email protected]

NUMUNRESPNODES=`curl ${UNRESPURL} 2> /dev/null | grep -Po '(?<=<a href="/nodes/unresponsive">)([^</a>]*)' | grep -v "Unresponsive"`
NUMERRORNODES=`curl ${ERRORURL} 2> /dev/null | grep -Po '(?<=<a href="/nodes/failed">)([^</\a>]*)' | grep -v "Failed"`

# Check if there are any unresponsive nodes, before emailing
if [ ${NUMUNRESPNODES} -gt 0 ]; then
        # Email an alert
        echo "There are ${NUMUNRESPNODES} puppet nodes that haven't reported back to the puppetmaster. Please check ${UNRESPURL} for more info." | mail -s "ALERT: ${NUMUNRESPNODES} puppet nodes are unresponsive" ${EMAIL}
fi

# Check if there are any nodes with errors, before emailing
if [ ${NUMERRORNODES} -gt 0 ]; then
    # Email an alert
    echo "There are ${NUMERRORNODES} puppet nodes that are reporting errors. Please check ${ERRORURL} for more info." | mail -s "ALERT: ${NUMERRORNODES} puppet nodes are reporting errors" ${EMAIL}
fi

exit 0
1
  • There are errors on -gt 0 on both lines. How does the script gain access to the dashboard?
    – nicoX
    Jan 30, 2015 at 17:35
0

What we currently do at Stack Exchange (Saw some upvotes on this question so figured I would add a updated answer):

Our monitoring agent scollector reads the run summary file in /var/lib/puppet/state/last_run_summary.yaml. When converted to a struct the relevant porition of that file looks as follows:

Resources struct {
    Changed         float64 `yaml:"changed"`
    Failed          float64 `yaml:"failed"`
    FailedToRestart float64 `yaml:"failed_to_restart"`
    OutOfSync       float64 `yaml:"out_of_sync"`
    Restarted       float64 `yaml:"restarted"`
    Scheduled       float64 `yaml:"scheduled"`
    Skipped         float64 `yaml:"skipped"`
    Total           float64 `yaml:"total"`
} `yaml:"resources"`

The agent then sends the "Failed" resource count ) which acts as a gauge. I can then have a rule in our monitoring system bosun that alerts if there have been failed resources consistently for a failed period of time:

$t = 2h
$notes = This alert will trigger when a host has consistently had failed resources for longer than $t
$q = min(q("sum:1h-min:puppet.run.resources{resource=failed,host=*}", "$t", ""))
warn = $q

This method of getting the stats from last_run_summary.yaml should be applicable to any monitoring system really.

0

I'm looking to get the Log>Level>err message, which as well can create an err message on a successful run. This is generated by the log file, and can be seen in the dashboard:

err Failed to call refresh: Could not restart Service[haproxy]: Execution of '/sbin/service haproxy restart' returned 1:

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .