How do you monitor a monitoring server?

Question

So we run Groundworks (with Nagios) on CentOS to monitor our various servers and processes. I have it setup to automatically send emails and SMS texts when things reach a WARNING or CRITICAL state. Normally this works perfectly. However, twice we've had problems with Postfix on that server where Postfix decides to stop sending email. The most recent time lasted 4 days because none of us noticed.

That leads me to a important question: how am I supposed to monitor my monitoring server?

Answer 1

With a second monitoring server, of course. The second one can be much simpler, since all it needs to do is monitor the first. And it should be monitored by the main monitoring system in turn, of course.

If your group is part of a larger organization with separate IT infrastructures, you may be able to make arrangements for another group's monitoring service to watch yours.

You could also make sure the server sends an "it's okay" message every day, and get in the habit of looking for it. (That's only effective if you're not already overwhelmed with routine messages, of course.)

Answer 2

Other people suggest sending out regular messages saying things are ok, but personally I don't agree with that. Monitoring should be silent unless there's a problem, and should never rely on a user noticing that something is wrong, like "Oh, I haven't gotten that daily e-mail in a few days." Especially if you have more than one person responding to alerts, each may think that the other has already removed the daily "I'm ok" message.

We have an external service (of which there are hundreds, but we use wormly) to do HTTP checks of our monitoring server to make sure that it's up and can reach the Internet. That's our primary concern for monitoring it. Then our Nagios server monitors all of our clients Nagios servers.

But, you bring up a good point. We probably should add an HTTP URL that checks the postfix queue and if it shows unusual number of messages, which probably means it has any in the queue, then raise an alert. Another option would be to use different methods for alerts, say a non-SMTP SMS delivery agent as well as SMTP that we currently use.

In our case though, I can't recall that we've ever had the mail server die. Of course, all that mail server is used for is sending Nagios alerts, so the configuration is very simple and almost never changes.

Answer 3

Obviously your postfix should be monitored too, but thats another topic ;)

I use Nagios checker plugin for Firefox, it is always running in a status bar on any computer I use regularly.

In addition I have a custom script on the outside host that pings the nagios host and sends SMS if its not responding to pings.

So far (5+ years) it worked ok (knock on wood).

Answer 4

For monitoring server monitoring (nagios in our case), the free or basic plan of Pingdom or alertfox works great.

Answer 5

First thing: Let it send "I am alive" messages once or twice a day. Second, I run an old machine just for this purpose, which has another GSM modem, a small UPS etc. and a dedicated (direct) connection to the primary monitoring server. This one helps with point three as well: Make sure you check the status of your monitoring systems regularly. The small auxiliary monitoring system displays the status page of the primary system in my office all the time.

Answer 6

If your Monitoring server is reachable from the internet you should have it monitored by external provider (e.g. websitepulse et. al.).