Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am researching about the most proficient way of administering 20 Linux servers and 100 Linux Workstations on a centralized basis.

I am not sure if there is some Administration and Monitoring Suite to achieve the daily administration work and troubleshooting from one single station.

share|improve this question
add comment

4 Answers

I have one site with forty workstations and about fifteen compute nodes.

I manage the workstations by:

  • forcing engineers to store all data on the NFS network, not locally
  • not letting any engineer have root on any workstation for any reason
  • having all systems syslog to a central syslog-ng host, with log parsing happening at regular intervals (usually daily, but sometimes as frequently as hourly)
  • monitor up/down with nagios
  • having a repeatable kickstart environment -- rule of thumb is if a problem can't be fixed in thirty minutes, the machine gets re-kickstarted (in practice, we actually kickstart much faster than that because in this setup there's rarely thirty minutes of troubleshooting we can do), and if the kickstart fails we start swapping hardware

I manage the compute farms pretty much the same way, except:

  • there is a local /scratch directory where anyone can write anything -- however the contents of that directory are not guaranteed
  • performance/usage counters are done through munin from a central host
  • network activity is done by using cacti to monitor the switch ports the farm nodes are connected to

It isn't perfect, but it has kept this one site going.

(Oh, I should mention that this site doesn't have any on-site full-time IT people, it is PT and on-demand. The monitoring systems above can usually let you know when there's a computer in distress.)

share|improve this answer
    
Nice. I smell infrastructures.org, no? –  pboin Nov 20 '09 at 16:52
    
No, actually -- I had never heard of it before your comment. Just the result of years of doing things the hard way and eventually learning. But I'll probably read through the infrastructures.org site and take the best ideas from it. :) –  David Mackintosh Nov 21 '09 at 1:27
add comment

On the automation from you have several options including

On the monitoring side I would suggest either Icinga or Nagios which are basically identical.

Hope this helps. The real thing to do is plan out just what you want to use the automation and monitoring for and then chose the best solution based on your requires. Everything has its own advantages and disadvantages, so carefully plan and select what you want.

share|improve this answer
add comment

I get the most value out of using chef to manage my servers' configuration. Monit, SEC, Collectd, and Icinga help me monitor them.

share|improve this answer
add comment

What are your "daily administration" tasks? What distro are you running?

Logwatch and cron emails go a long way to helping you keep abreast of what's going on on your servers. Beyond that, products like Puppet can help automate things like pushing out packages, changing firewall configs, etc. There are also tools like Parallel SSH that can help you manually execute certain command lines on a bunch of servers at once.

If you're running a debian-based solution, there is a package called apticron that will run periodically via cron and can email you a report of when there are packages that need updating. I know there is a similar program for Redhat/CentOS type distros, but the name is escaping me currently.

As far as monitoring goes, there are a vast number of packages that you could check out. I'm rather partial to Nagios, but there are many free and for-pay systems out there that can do a good job. Without giving more details, though, it's quite difficult for us to give good recommendations for your specific situation.

share|improve this answer
1  
apt-dater is also a useful Debian solution for managing package upgrades. –  Haakon Nov 20 '09 at 8:24
    
For RedHat based distros yum-updatesd will report hosts, that need updating. –  sumar Nov 20 '09 at 9:51
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.