up vote 0 down vote favorite
share [g+] share [fb]

I was in a dicussion with a colleague today and I mentioned how servers are not proactively monitored in a manual way (i.e. rdp to random servers, and do some manual checks - this would go in conjunction with proper, automated monitoring, but we do not have that either).

As a best practise in system and server administration - do you recommend checking servers manually every day in conjunction with automated monitoring?

link|improve this question
30% accept rate
2  
I kinda wonder if you are looking for a Chaos Monkey. – Zoredache Jan 17 at 22:13
feedback

4 Answers

up vote 4 down vote

As a best practise in system and server administration - do you recommend checking servers manually every day in conjunction with automated monitoring?

In short, no.

Your monitoring system must be comprehensive enough to monitor every critical service provided by each system.

Why? Manual monitoring just plain does not scale. If you have 5 servers, it's one thing, but what are you going to do when you scale that to 50 or 500? You don't want to pay your sysadmins to go through and poke a huge list of servers each day.

link|improve this answer
But how are you going to be sure your monitor isn't giving you a false positive? – Zoredache Jan 17 at 22:15
2  
Well, of course some sanity checking is warranted from time to time. Something is wrong, though, if you have so little confidence in your monitoring system that you feel obligated to double-check things every day. – ErikA Jan 17 at 22:21
feedback
up vote 2 down vote

I would absolutely not recommend that - other than as an exercise to identify where your automated monitoring falls short.

Why? Simple - people get bored, and forget things. Machines don't. So, if you manually check something to make sure the server is working, write a script to do the same thing, and have the server invoke it routinely.

link|improve this answer
"People get bored, and forget things" Excellent point. – ErikA Jan 17 at 22:07
feedback
up vote 2 down vote

If your software monitoring solution is working well, I don't see a reason to do "spot checks". That said, I do at least one lap a day around our data center to see if there are any amber lights blinking that monitors might have missed. If you colo, this probably isn't feasible. For me, it only takes 5 minutes or so out of my day.

link|improve this answer
feedback
up vote -1 down vote

I also don't really see the point in manually checking stuff. It doesn't scale and it would seem cumbersome to do.

Something that is of great importance imo though is actually thinking about what exactly to monitor. Like there is no need to monitor every single service on a machine - its just wasted traffic and pointlessly added load. Keep it down to the important stuff. Disks, Raids, those one or two important services and most importantly logfiles. I tend to think that doing a good job at optimizing logfile auditing saves you a shitload of work and time afterwards

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.