The short version: I've got a heterogeneous environment of ~400 hosts using Groundwork/Nagios for monitoring. The current checks, hostgroups, and servicegroups have been put together in an organic, ad-hoc manner. I'm tasked with essentially rebuilding the monitoring setup.
My previous gig involved less than 20 machines with no strict after-hours uptime requirements, monitored with Munin - this is beyond my experience. I am, at base, looking for a process by which this task can be tackled.
I have a vague notion of designing high level end-to-end checks for end user services - stuff like a scraper attempting to log in to one of our websites - and then have a bevy of more specific, standard checks set up as dependent checks - stuff like checking that httpd is running, the host is available via the network, on down the stack - and only having the lower level checks run when the high level checks fail as a way of providing visibility into the root cause while minimizing system strain. I also am thinking generally of dividing hosts up by environment so that the team only gets pages from production boxes after hours, that sort of thing.
Is this sane? Is there a best-practices approach for designing a monitoring system? I'm confident in my ability to migrate from our current less than ideal setup to something better designed, but I'd like some more seasoned guidance on how to design an ideal setup in the first place.