How do you administer 20 or more Linux Servers daily?

Question

I am researching about the most proficient way of administering 20 Linux servers and 100 Linux Workstations on a centralized basis.

I am not sure if there is some Administration and Monitoring Suite to achieve the daily administration work and troubleshooting from one single station.

Answer 1

I have one site with forty workstations and about fifteen compute nodes.

I manage the workstations by:

• forcing engineers to store all data on the NFS network, not locally
• not letting any engineer have root on any workstation for any reason
• having all systems syslog to a central syslog-ng host, with log parsing happening at regular intervals (usually daily, but sometimes as frequently as hourly)
• monitor up/down with nagios
• having a repeatable kickstart environment -- rule of thumb is if a problem can't be fixed in thirty minutes, the machine gets re-kickstarted (in practice, we actually kickstart much faster than that because in this setup there's rarely thirty minutes of troubleshooting we can do), and if the kickstart fails we start swapping hardware

I manage the compute farms pretty much the same way, except:

• there is a local /scratch directory where anyone can write anything -- however the contents of that directory are not guaranteed
• performance/usage counters are done through munin from a central host
• network activity is done by using cacti to monitor the switch ports the farm nodes are connected to

It isn't perfect, but it has kept this one site going.

(Oh, I should mention that this site doesn't have any on-site full-time IT people, it is PT and on-demand. The monitoring systems above can usually let you know when there's a computer in distress.)

Answer 2

On the automation from you have several options including

• cfengine
• puppet (currently what I use)
• chef

On the monitoring side I would suggest either Icinga or Nagios which are basically identical.

Hope this helps. The real thing to do is plan out just what you want to use the automation and monitoring for and then chose the best solution based on your requires. Everything has its own advantages and disadvantages, so carefully plan and select what you want.

Answer 3

I get the most value out of using chef to manage my servers' configuration. Monit, SEC, Collectd, and Icinga help me monitor them.

Answer 4

What are your "daily administration" tasks? What distro are you running?

Logwatch and cron emails go a long way to helping you keep abreast of what's going on on your servers. Beyond that, products like Puppet can help automate things like pushing out packages, changing firewall configs, etc. There are also tools like Parallel SSH that can help you manually execute certain command lines on a bunch of servers at once.

If you're running a debian-based solution, there is a package called apticron that will run periodically via cron and can email you a report of when there are packages that need updating. I know there is a similar program for Redhat/CentOS type distros, but the name is escaping me currently.

As far as monitoring goes, there are a vast number of packages that you could check out. I'm rather partial to Nagios, but there are many free and for-pay systems out there that can do a good job. Without giving more details, though, it's quite difficult for us to give good recommendations for your specific situation.