1

I am managing more than 100 servers and I have some package that needs to be updated. Currently, I just have a shell script running a for loop on all servers and executing the command.

Does anyone know the best (or a better) way to do this?

5 Answers 5

5

If you're running 100+ servers, you really should look into a configuration management system, such as cfengine, puppet, bcfg2, etc. A well-configured system would be able to push out routine package upgrades very easily, with little or no additional scripting.

2
3

What I did was to create my own internal repo (which is painfully simple):

  1. Create a web accessible directory
  2. Put RPMs there
  3. Run "createrepo ./"
  4. Point your clients there by adding the following file into your /etc/yum.repos.d/ directory:
name=My Repo
baseurl=http://path/to/directory/above
gpgcheck=0

You can now install RPMs from the directory that you made.

Note that this solution has no inherent security...it doesn't check the signed status of the packages. The steps to set up the gpg signatures aren't hard, and are detailed at this howto: http://mindtrove.info/articles/hosting-a-yum-repository/

2

We have a similarly developed tool (written in python here, in perl at my previous job) tool to Aleksandr, and that approach tends to work out fine.

I've seen a tool that's designed for just this kind of thing that has captured my interest, TakTuk but I haven't had enough time to buckle down and start using it. It's designed with scalability in mind, so if your environment is likely to keep on growing, you might want to take a look at it. Capistrano might also be worth investigating.

2

Spacewalk is a the upstream FOSS version of Red Hat Satellite Server. You might could give that a try if you need more robust features than can by provided by a single script.

Spacewalk: Free & Open Source Linux Systems Management

2

Instead of a for-loop you can execute the commands in parallel and then collect the output sequentially in the ordered that you want. I wrote the following Ruby script and used it for several years for massive package installs, quick configuration changes, and other tasks. It speed things up a lot:

Script: on-all-nodes-run

#!/usr/bin/ruby

EXCEPT = [] 

require "open3"

SSH_OPTIONS = ["-o PreferredAuthentications=hostbased,gssapi,gssapi-with-mic",
               "-o ForwardX11=no",
               "-o BatchMode=yes",
               "-o SetupTimeOut=5",
               "-o ServerAliveInterval=5",
               "-o ServerAliveCountMax=2"
              ].join(" ")

SSH    = "/usr/bin/ssh #{SSH_OPTIONS}"
MKDIR  = "/bin/mkdir"

raise "give this command at least one argument" if ARGV.size < 1
COMMAND = ARGV[0..-1].join(' ')

output_o = {}
output_e = {}

FORMAT = "%Y-%m-%d_%H-%M-%S_#{(rand * 100).to_i}"

IO_CONNECTIONS_TO_REMOTE_PROCESSES = {}

def on_all_nodes(&block)
  1.upto(32) do |i| 
    next if EXCEPT.include? i
    node = "node#{i.to_s.rjust(2, '0')}"
    block.call(node)
  end
end


# Create processes
on_all_nodes do |node|
  stdin, stdout, stderr = Open3.popen3("#{SSH} #{node} \"#{COMMAND}\"")
  IO_CONNECTIONS_TO_REMOTE_PROCESSES[node] = [stdin, stdout, stderr]
end


has_remote_errors = false

# Collect results
on_all_nodes do |node|
  stdin, stdout, stderr = IO_CONNECTIONS_TO_REMOTE_PROCESSES[node]

  stdin.close

  e_thread = Thread.new do
    while line = stderr.gets
      line.chomp!
      STDERR.puts "#{node} ERROR: #{line}"
      has_remote_errors = true
    end
  end

  o_thread = Thread.new do
    first = true
    while line = stdout.gets
      line.chomp!
      puts "#{node}      : #{line}"
    end
  end

  # Let the threads finish
  t1 = nil
  t2 = nil
  while [t1, t2].include? nil
    if t1.nil?
      t1 = e_thread.join(0.1) # Gives 1/10 of a second to STDERR
    end
    if t2.nil?
      t2 = o_thread.join(0.1) # Give 1/10 of a second to STDOUT
    end
  end

end


exit(1) if has_remote_errors

Miscellaneous Remarks

The script above relies on a standard naming pattern for the hosts. It's node01 node02 ... node99. If your hosts are all named differently then you can make aliases in /etc/hosts that are uniform.

Because you have more servers then me you would need to adjust my script to use something like: n001 n002 ... n100.

After installing a package run a check. I use Debian, but on a CentOS/RedHat system it would be something like on-all-nodes-run yum info <package-name>

I use on-all-nodes-run several times a month, but at some point a node will inevitably be down and will end-up out-of-date. So on top of on-all-nodes-run I re-image the nodes when there is chance to reboot (SystemImager, possibly KickStart, ...). Entire clsuter is re-imaged a few times pre year and every time a failed out-of-date node is rebooted.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .