The short version: I've got a heterogeneous environment of ~400 hosts using Groundwork/Nagios for monitoring. The current checks, hostgroups, and servicegroups have been put together in an organic, ad-hoc manner. I'm tasked with essentially rebuilding the monitoring setup.

My previous gig involved less than 20 machines with no strict after-hours uptime requirements, monitored with Munin - this is beyond my experience. I am, at base, looking for a process by which this task can be tackled.

I have a vague notion of designing high level end-to-end checks for end user services - stuff like a scraper attempting to log in to one of our websites - and then have a bevy of more specific, standard checks set up as dependent checks - stuff like checking that httpd is running, the host is available via the network, on down the stack - and only having the lower level checks run when the high level checks fail as a way of providing visibility into the root cause while minimizing system strain. I also am thinking generally of dividing hosts up by environment so that the team only gets pages from production boxes after hours, that sort of thing.

Is this sane? Is there a best-practices approach for designing a monitoring system? I'm confident in my ability to migrate from our current less than ideal setup to something better designed, but I'd like some more seasoned guidance on how to design an ideal setup in the first place.

  • 3
    You might find this blog post of mine helpful as to answering "What to montior" in a structured way: blog.serverfault.com/post/747186396 – Kyle Brandt Jan 21 '11 at 19:40
  • Kyle's post is excellent. Note that you don't have to go all the way down the stack - You can monitor for the orange circles (stuff you can proactively fix) and a selection of blue triangles (pre-failure conditions like 90% of RAM and 90% of Swap in use), as well as obviously the major failure condition of "Website Down". That's fairly comprehensive and reasonable to implement. – voretaq7 Jan 21 '11 at 19:46
  • @Vortaq7: Yup, I view it as more of a brainstorming tool unless you have extreme cases of uptime. In which case those sort of analysis are probably someones full time job. – Kyle Brandt Jan 21 '11 at 20:06
  • It's worth noting that any answers that pop up on here are probably equally applicable to Windows environments. The same thought process applies no matter what you're monitoring. – voretaq7 Jan 21 '11 at 20:11
  • 1
    The link to Kyle's blog post is broken :-( – dr-jan Jan 27 '12 at 16:37

To expand on my comment and hopefully give you a little guidance, what you probably want to take away from Kyle's blog post (and this one too, both on my list of must-read references for people designing monitoring systems) is that a failure isn't usually when one thing goes wrong - it's when 10 things go wrong.
The job of a decent monitoring system is to catch those 10 things before they actually take down your services and impact client-facing stuff.

What follows below is by no means exhaustive or complete, but is pretty similar to my method of tackling monitoring setup & should get you going in the right direction:

  1. To figure out what you want to monitor you first need to think about what can cause stuff to fail.

    1. Some of these things are common
      Many of these can be cribbed from Kyle's post so I won't list them, but you want to get notified about PRE-FAILURE conditions - e.g. ONE drive in a RAID5 failed -- replace it now and avoid the downtime later.
    2. Others vary based on your infrastructure/design & include dependencies on other services
      If you're running a database-backed web site and the database is down your site won't work
  2. Take a look at the dependencies and build a dependency tree.
    (In a datacenter you manage this can go as far as you like: At my last job we were a hosting company, and our monitoring system talked to our UPS, Generator and Cooling systems to keep us up to date on their status)

  3. Armed with all of that information you can decide what can be proactively monitored and what can only be reactively responded to
    (e.g. "Network cable(s) yanked out" will bring any server down, but is it worth monitoring the switch port's status, or do you want that to be a "It's down, I have to go look at it" situation?).

That only leaves the task of selecting monitoring software to implement your vision. This selection should be based on which package lets you monitor all (or most) of what your ideal list is, and realistically will take in to account price differences between the packages.

  • The link to Kyle's blog post is broken. – dr-jan Jan 27 '12 at 16:38
  • @dr-jan you're right -- apparently the link structure within the ServerFault blog has changed since last year (grumble) -- the whole category of "monitoring stuff" is now viewable though (Look here). Peter actually just published another article on monitoring today, which is also an excellent read. – voretaq7 Jan 27 '12 at 21:54
  • I think this is a new link to that post. – rschwieb Jun 18 '15 at 20:34
  • @rschwieb Actually I think it was a different one, but I found the one I was thinking of (and linked to both in the answer). Only took me 4 years to quit being lazy and fix that link rot - I'm a terrible moderator :P – voretaq7 Jun 18 '15 at 20:44
  • 1
    @voretaq7 I'm pretty sure the link I posted is correct: I used the Wayback machine on the dead link and found the article as it appeared in 2011. Thanks for posting the other one, though: it looks just as helpful. – rschwieb Jun 18 '15 at 20:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.