What is a solid process for designing a monitoring system?

Question

The short version: I've got a heterogeneous environment of ~400 hosts using Groundwork/Nagios for monitoring. The current checks, hostgroups, and servicegroups have been put together in an organic, ad-hoc manner. I'm tasked with essentially rebuilding the monitoring setup.

My previous gig involved less than 20 machines with no strict after-hours uptime requirements, monitored with Munin - this is beyond my experience. I am, at base, looking for a process by which this task can be tackled.

I have a vague notion of designing high level end-to-end checks for end user services - stuff like a scraper attempting to log in to one of our websites - and then have a bevy of more specific, standard checks set up as dependent checks - stuff like checking that httpd is running, the host is available via the network, on down the stack - and only having the lower level checks run when the high level checks fail as a way of providing visibility into the root cause while minimizing system strain. I also am thinking generally of dividing hosts up by environment so that the team only gets pages from production boxes after hours, that sort of thing.

Is this sane? Is there a best-practices approach for designing a monitoring system? I'm confident in my ability to migrate from our current less than ideal setup to something better designed, but I'd like some more seasoned guidance on how to design an ideal setup in the first place.

Answer 1

To expand on my comment and hopefully give you a little guidance, what you probably want to take away from Kyle's blog post (now on my list of must-read references for people designing monitoring systems) is that a failure isn't usually when one thing goes wrong - it's when 10 things go wrong.
The job of a decent monitoring system is to catch those 10 things before they actually take down your services and impact client-facing stuff.

What follows below is by no means exhaustive or complete, but is pretty similar to my method of tackling monitoring setup & should get you going in the right direction:

---

1. To figure out what you want to monitor you first need to think about what can cause stuff to fail.

   1. Some of these things are common
      Many of these can be cribbed from Kyle's post so I won't list them, but you want to get notified about PRE-FAILURE conditions - e.g. ONE drive in a RAID5 failed -- replace it now and avoid the downtime later.
   2. Others vary based on your infrastructure/design & include dependencies on other services
      If you're running a database-backed web site and the database is down your site won't work
       

2. Take a look at the dependencies and build a dependency tree.
   (In a datacenter you manage this can go as far as you like: At my last job we were a hosting company, and our monitoring system talked to our UPS, Generator and Cooling systems to keep us up to date on their status)

3. Armed with all of that information you can decide what can be proactively monitored and what can only be reactively responded to
   (e.g. "Network cable(s) yanked out" will bring any server down, but is it worth monitoring the switch port's status, or do you want that to be a "It's down, I have to go look at it" situation?).

That only leaves the task of selecting monitoring software to implement your vision. This selection should be based on which package lets you monitor all (or most) of what your ideal list is, and realistically will take in to account price differences between the packages.