Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I was in a dicussion with a colleague today and I mentioned how servers are not proactively monitored in a manual way (i.e. rdp to random servers, and do some manual checks - this would go in conjunction with proper, automated monitoring, but we do not have that either).

As a best practise in system and server administration - do you recommend checking servers manually every day in conjunction with automated monitoring?

share|improve this question
2  
I kinda wonder if you are looking for a Chaos Monkey. –  Zoredache Jan 17 '12 at 22:13

4 Answers 4

As a best practise in system and server administration - do you recommend checking servers manually every day in conjunction with automated monitoring?

In short, no.

Your monitoring system must be comprehensive enough to monitor every critical service provided by each system.

Why? Manual monitoring just plain does not scale. If you have 5 servers, it's one thing, but what are you going to do when you scale that to 50 or 500? You don't want to pay your sysadmins to go through and poke a huge list of servers each day.

share|improve this answer
    
But how are you going to be sure your monitor isn't giving you a false positive? –  Zoredache Jan 17 '12 at 22:15
2  
Well, of course some sanity checking is warranted from time to time. Something is wrong, though, if you have so little confidence in your monitoring system that you feel obligated to double-check things every day. –  EEAA Jan 17 '12 at 22:21

I would absolutely not recommend that - other than as an exercise to identify where your automated monitoring falls short.

Why? Simple - people get bored, and forget things. Machines don't. So, if you manually check something to make sure the server is working, write a script to do the same thing, and have the server invoke it routinely.

share|improve this answer
    
"People get bored, and forget things" Excellent point. –  EEAA Jan 17 '12 at 22:07

If your software monitoring solution is working well, I don't see a reason to do "spot checks". That said, I do at least one lap a day around our data center to see if there are any amber lights blinking that monitors might have missed. If you colo, this probably isn't feasible. For me, it only takes 5 minutes or so out of my day.

share|improve this answer

I also don't really see the point in manually checking stuff. It doesn't scale and it would seem cumbersome to do.

Something that is of great importance imo though is actually thinking about what exactly to monitor. Like there is no need to monitor every single service on a machine - its just wasted traffic and pointlessly added load. Keep it down to the important stuff. Disks, Raids, those one or two important services and most importantly logfiles. I tend to think that doing a good job at optimizing logfile auditing saves you a shitload of work and time afterwards

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.