Do you monitor servers manually as well in conjunction to automated monitoring?

Question

I was in a dicussion with a colleague today and I mentioned how servers are not proactively monitored in a manual way (i.e. rdp to random servers, and do some manual checks - this would go in conjunction with proper, automated monitoring, but we do not have that either).

As a best practise in system and server administration - do you recommend checking servers manually every day in conjunction with automated monitoring?

Answer 1

I also don't really see the point in manually checking stuff. It doesn't scale and it would seem cumbersome to do.

Something that is of great importance imo though is actually thinking about what exactly to monitor. Like there is no need to monitor every single service on a machine - its just wasted traffic and pointlessly added load. Keep it down to the important stuff. Disks, Raids, those one or two important services and most importantly logfiles. I tend to think that doing a good job at optimizing logfile auditing saves you a shitload of work and time afterwards

Answer 2

If your software monitoring solution is working well, I don't see a reason to do "spot checks". That said, I do at least one lap a day around our data center to see if there are any amber lights blinking that monitors might have missed. If you colo, this probably isn't feasible. For me, it only takes 5 minutes or so out of my day.

Answer 3

As a best practise in system and server administration - do you recommend checking servers manually every day in conjunction with automated monitoring?

In short, no.

Your monitoring system must be comprehensive enough to monitor every critical service provided by each system.

Why? Manual monitoring just plain does not scale. If you have 5 servers, it's one thing, but what are you going to do when you scale that to 50 or 500? You don't want to pay your sysadmins to go through and poke a huge list of servers each day.

Answer 4

I would absolutely not recommend that - other than as an exercise to identify where your automated monitoring falls short.

Why? Simple - people get bored, and forget things. Machines don't. So, if you manually check something to make sure the server is working, write a script to do the same thing, and have the server invoke it routinely.