Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

So we run Groundworks (with Nagios) on CentOS to monitor our various servers and processes. I have it setup to automatically send emails and SMS texts when things reach a WARNING or CRITICAL state. Normally this works perfectly. However, twice we've had problems with Postfix on that server where Postfix decides to stop sending email. The most recent time lasted 4 days because none of us noticed.

That leads me to a important question: how am I supposed to monitor my monitoring server?

share|improve this question
5  
Quis custodiet ipsos custodes? –  James Lawrie Dec 10 '10 at 22:53
    
Heh. Juvenal. Nicely played. –  organicveggie Dec 11 '10 at 4:18
    
Who Watch the Watchmen ? :D –  Florent Courtay Dec 22 '10 at 15:10

6 Answers 6

up vote 13 down vote accepted

With a second monitoring server, of course. The second one can be much simpler, since all it needs to do is monitor the first. And it should be monitored by the main monitoring system in turn, of course.

If your group is part of a larger organization with separate IT infrastructures, you may be able to make arrangements for another group's monitoring service to watch yours.

You could also make sure the server sends an "it's okay" message every day, and get in the habit of looking for it. (That's only effective if you're not already overwhelmed with routine messages, of course.)

share|improve this answer

If your Monitoring server is reachable from the internet you should have it monitored by external provider (e.g. websitepulse et. al.).

share|improve this answer

For monitoring server monitoring (nagios in our case), the free or basic plan of Pingdom or alertfox works great.

share|improve this answer
    
Good suggestions. But in this case, our monitoring server is not accessible outside the firewall. So Pingdom and Alertfox don't really work for us. –  organicveggie Dec 11 '10 at 4:22

Other people suggest sending out regular messages saying things are ok, but personally I don't agree with that. Monitoring should be silent unless there's a problem, and should never rely on a user noticing that something is wrong, like "Oh, I haven't gotten that daily e-mail in a few days." Especially if you have more than one person responding to alerts, each may think that the other has already removed the daily "I'm ok" message.

We have an external service (of which there are hundreds, but we use wormly) to do HTTP checks of our monitoring server to make sure that it's up and can reach the Internet. That's our primary concern for monitoring it. Then our Nagios server monitors all of our clients Nagios servers.

But, you bring up a good point. We probably should add an HTTP URL that checks the postfix queue and if it shows unusual number of messages, which probably means it has any in the queue, then raise an alert. Another option would be to use different methods for alerts, say a non-SMTP SMS delivery agent as well as SMTP that we currently use.

In our case though, I can't recall that we've ever had the mail server die. Of course, all that mail server is used for is sending Nagios alerts, so the configuration is very simple and almost never changes.

share|improve this answer
2  
Regular OK messages aren't that useful: you cannot reliably condition a person to make an action in the absence of a stimulus. –  Tim Williscroft Dec 5 '10 at 23:02
    
@Tim: Sorry, but "absence of a stimulus" does not describe the situation where an anticipated email is not received. In such a case, I believe I would be "stimulated" to investigate why the message did not arrive. But maybe that's just me. :) –  Steven Monday Dec 6 '10 at 0:06
1  
I think I'm writing using psychological terms that don't mean what you think they mean. Behavioural psychology, and aviation psychology have a lot to tell system engineers. The field was developed heavily in WWII to get 18-20 year old crews to fly state-of-the art aircraft without crashing, and to still have attention left over for their real military tasks. That's why aircraft have a master caution light, not a "everything is okay" light. TLDR (I don't think that word means what you think it means) –  Tim Williscroft Dec 6 '10 at 1:09
1  
I'm very strongly of the opinion that systems shouldn't make noise unless there is something that needs attention from a human. We have finite attention, and computers can easily overwhelm us with little blips like "I'm alive!". Plus, things that come through that don't indicate problems put people in the mind of ignoring things. I work very hard to make sure that when something comes to a human, it is something they really need to see. I work with someone who has all sorts of logs coming to him every day that he reviews. Of course, he's so busy he can't go out to lunch... –  Sean Reifschneider Dec 6 '10 at 20:47
1  
I agree that services shouldn't send too many messages or people rapidly start ignoring them. However, if the monitoring system is setup correctly, you shouldn't be getting a lot of messages. In course, we have a policy about acknowledging alerts from Groundworks/Nagios, which effectively stops the messages for a period of time. If it's a long term outage, we disable the monitoring for the system or service. As a result, a daily "I'm Alive" message is actually pretty reasonable. –  organicveggie Dec 11 '10 at 4:25

First thing: Let it send "I am alive" messages once or twice a day. Second, I run an old machine just for this purpose, which has another GSM modem, a small UPS etc. and a dedicated (direct) connection to the primary monitoring server. This one helps with point three as well: Make sure you check the status of your monitoring systems regularly. The small auxiliary monitoring system displays the status page of the primary system in my office all the time.

share|improve this answer

Obviously your postfix should be monitored too, but thats another topic ;)

I use Nagios checker plugin for Firefox, it is always running in a status bar on any computer I use regularly.

In addition I have a custom script on the outside host that pings the nagios host and sends SMS if its not responding to pings.

So far (5+ years) it worked ok (knock on wood).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.