Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have a pretty simple system setup, where I get text messages when there is a system problem. It's nothing fancy. I send an email to my phone number within my logging class for alert levels. It works well enough, but it has one major flaw: A small hiccup in the system/site can turn into dozens of rapid fire text messages. Sometimes non-stop text messages until I log into the system and fix the problem.

So I'm looking for pointers on software or services I can use that deal with alerts in a smarter way. Perhaps something that only sends me alerts X number of times within Y number of minutes. I'm not looking for a full monitoring suite. We already deal with that in house. I'm only looking to tackle this single problem.

share|improve this question
    
What monitoring system are you currently using? –  JakeRobinson Feb 21 '11 at 5:27
    
@Jake - It's hard to say. I'm not the sysadmin, I'm the web guy. I only came here because our sysadmin has been blowing me off on giving me a hand with this problem. I personally use my own home brewed scripts that monitor the site and the back end services it relies on, and I send out alerts through my logging class when something is wrong. I'm not sure what he uses to monitor the network and servers. –  mellowsoon Feb 21 '11 at 7:07

2 Answers 2

up vote 2 down vote accepted

I create small bash script for you:

#!/bin/bash

COUNT_FILE="/tmp/count"
TIME_FILE="/tmp/time"
MAX_SEND=1             #max message
TIME_INT=300            #time interval in second


send () {
        ERROR_TYPE="_$1"
        MESSAGE=$2
        [ -e ${TIME_FILE}${ERROR_TYPE} ] || touch ${TIME_FILE}${ERROR_TYPE}
        [ -e ${COUNT_FILE}${ERROR_TYPE} ] || echo 0 > ${COUNT_FILE}${ERROR_TYPE}
        if [ $(( $(date +%s) - $(date +%s -r ${TIME_FILE}${ERROR_TYPE}) )) -gt $TIME_INT ];
        then
                COUNT=0
                touch ${TIME_FILE}${ERROR_TYPE}
        else
                COUNT=`cat ${COUNT_FILE}${ERROR_TYPE}`
        fi
        if [ $COUNT -lt $MAX_SEND ];
        then
                echo "$MESSAGE";
                #real send message
        fi
        COUNT=$(($COUNT+1))
        echo $COUNT > "${COUNT_FILE}${ERROR_TYPE}"
}

send "check_dns" "message"
share|improve this answer
    
Wow. So I assume it would send a max of 10 messages in 200 seconds? –  mellowsoon Feb 21 '11 at 7:02
    
@mellowsoon Change to one messages in 300 second –  ooshro Feb 21 '11 at 7:22
    
@ooshro, thanks, it works like a charm. Glad you came up with this. I'm the kind of person that would have written a whole system involving databases and messaging queues to solve such a simple problem. –  mellowsoon Feb 21 '11 at 7:55
    
@ooshro, I had to take the answer back. This solution suffers from a few major flaws. 1) It crashes if either of the temp files doesn't exist. 2) It doesn't take into account the type of message. I might set it to send only 1 text in 10 minutes, however that should be per error type. Meaning I'd still get several texts within 10 minutes if they're different messages. –  mellowsoon Feb 24 '11 at 13:52
    
@mellowsoon I update script –  ooshro Feb 24 '11 at 23:38

The answer to such questions is most often Nagios. The alerting options are as flexible as it's monitoring capabilities. Configure it to send only the alerts you want and no more.

share|improve this answer
    
thanks. I'll look into it. –  mellowsoon Feb 24 '11 at 13:52
    
+1 for Nagios. There's a learning curve but you'll be rewarded for configuring it well. –  Nathan Feb 24 '11 at 23:48

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.