I am managing more than 100 servers and I have some package that needs to be updated. Currently, I just have a shell script running a for loop
on all servers and executing the command.
Does anyone know the best (or a better) way to do this?
If you're running 100+ servers, you really should look into a configuration management system, such as cfengine, puppet, bcfg2, etc. A well-configured system would be able to push out routine package upgrades very easily, with little or no additional scripting.
What I did was to create my own internal repo (which is painfully simple):
name=My Repo baseurl=http://path/to/directory/above gpgcheck=0
You can now install RPMs from the directory that you made.
Note that this solution has no inherent security...it doesn't check the signed status of the packages. The steps to set up the gpg signatures aren't hard, and are detailed at this howto: http://mindtrove.info/articles/hosting-a-yum-repository/
We have a similarly developed tool (written in python here, in perl at my previous job) tool to Aleksandr, and that approach tends to work out fine.
I've seen a tool that's designed for just this kind of thing that has captured my interest, TakTuk but I haven't had enough time to buckle down and start using it. It's designed with scalability in mind, so if your environment is likely to keep on growing, you might want to take a look at it. Capistrano might also be worth investigating.
Spacewalk is a the upstream FOSS version of Red Hat Satellite Server. You might could give that a try if you need more robust features than can by provided by a single script.
Instead of a for-loop you can execute the commands in parallel and then collect the output sequentially in the ordered that you want. I wrote the following Ruby script and used it for several years for massive package installs, quick configuration changes, and other tasks. It speed things up a lot:
#!/usr/bin/ruby
EXCEPT = []
require "open3"
SSH_OPTIONS = ["-o PreferredAuthentications=hostbased,gssapi,gssapi-with-mic",
"-o ForwardX11=no",
"-o BatchMode=yes",
"-o SetupTimeOut=5",
"-o ServerAliveInterval=5",
"-o ServerAliveCountMax=2"
].join(" ")
SSH = "/usr/bin/ssh #{SSH_OPTIONS}"
MKDIR = "/bin/mkdir"
raise "give this command at least one argument" if ARGV.size < 1
COMMAND = ARGV[0..-1].join(' ')
output_o = {}
output_e = {}
FORMAT = "%Y-%m-%d_%H-%M-%S_#{(rand * 100).to_i}"
IO_CONNECTIONS_TO_REMOTE_PROCESSES = {}
def on_all_nodes(&block)
1.upto(32) do |i|
next if EXCEPT.include? i
node = "node#{i.to_s.rjust(2, '0')}"
block.call(node)
end
end
# Create processes
on_all_nodes do |node|
stdin, stdout, stderr = Open3.popen3("#{SSH} #{node} \"#{COMMAND}\"")
IO_CONNECTIONS_TO_REMOTE_PROCESSES[node] = [stdin, stdout, stderr]
end
has_remote_errors = false
# Collect results
on_all_nodes do |node|
stdin, stdout, stderr = IO_CONNECTIONS_TO_REMOTE_PROCESSES[node]
stdin.close
e_thread = Thread.new do
while line = stderr.gets
line.chomp!
STDERR.puts "#{node} ERROR: #{line}"
has_remote_errors = true
end
end
o_thread = Thread.new do
first = true
while line = stdout.gets
line.chomp!
puts "#{node} : #{line}"
end
end
# Let the threads finish
t1 = nil
t2 = nil
while [t1, t2].include? nil
if t1.nil?
t1 = e_thread.join(0.1) # Gives 1/10 of a second to STDERR
end
if t2.nil?
t2 = o_thread.join(0.1) # Give 1/10 of a second to STDOUT
end
end
end
exit(1) if has_remote_errors
The script above relies on a standard naming pattern for the hosts. It's node01 node02 ... node99. If your hosts are all named differently then you can make aliases in /etc/hosts that are uniform.
Because you have more servers then me you would need to adjust my script to use something like: n001 n002 ... n100.
After installing a package run a check. I use Debian, but on a CentOS/RedHat system it would be something like on-all-nodes-run yum info <package-name>
I use on-all-nodes-run several times a month, but at some point a node will inevitably be down and will end-up out-of-date. So on top of on-all-nodes-run
I re-image the nodes when there is chance to reboot (SystemImager, possibly KickStart, ...). Entire clsuter is re-imaged a few times pre year and every time a failed out-of-date node is rebooted.