Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have many servers and still want to update them all. The actual way is that any of the sysadmins go from server to server and make a aptitude update && aptitude upgrade - it's still not cool.

I am searching now for a solution which is still better and very smart. Can puppet do this job? How do you do it?

share|improve this question
    
yes, puppet can do this. cssh would fix your problem in the short term, also. –  Sirex Jan 3 '12 at 7:43
add comment

11 Answers

up vote 10 down vote accepted

You can use the exec type such as:

exec { "upgrade_packages":
    command => "apt-get upgrade -q=2",
    path    => "/usr/local/bin/:/bin/:/usr/bin/",
    # path  => [ "/usr/local/bin/", "/bin/" ],  # alternative syntax
}

To be honest, I did not try it myself, but I think you just need to create a new module that include such an exec definition.

The apt-get upgrade command is interactive. To make it run quietly, you can add the option -q=2 as shown above.

share|improve this answer
    
looks very good! I think I create an Puppet Usecase with some Testmachines to try this. Many Thanks! –  Dennis Wisnia Jan 3 '12 at 8:34
3  
+1 for recommending Puppet ! Changes your life as a sysadmin :) –  Antoine Benkemoun Jan 3 '12 at 10:27
3  
Mind that this exec will run on every Puppet run (every 30 minutes) which might hammer your proxy and / or your mirror pretty hard if you have a "lot of servers". Personally, I would recommend implementing a schedule for the exec type above, making sure it only runs at night, for example. In my opinion though, Puppet is meant to enforce system state, and running an upgrade_packages-like command without human oversight through it, is both a bit scary and a bit of Puppet abuse. The mColective tool that comes with Puppet Enterprise (or it's open source equivalent) might be a better option. –  wzzrd Jan 3 '12 at 10:31
7  
Our puppet installation has a similar exec that checks the timestamp on a file and only runs the upgrade if it's newer than the version on the client. When we want to upgrade everything, we touch that file on the puppet master. –  Ladadadada Jan 3 '12 at 10:44
    
@wzzrd: Good point, but it can be made better by checking some external condition as Ladadadada said. –  Khaled Jan 3 '12 at 10:57
show 1 more comment

Another solution if all your hosts are running Debian (or derivatives) is to use the cron-apt package. But, as suggested per the documentation, a bit of care must be taken.

I'm currently using cron-apt on a dozen of servers to perform all the security updates automatically and unattended. To avoid any unwanted upgrades, I only use cron-apt on servers which runs the Debian stable distribution and I make sure to configure my apt sources so use the distribution name, wheezy, and not its alias (stable).

The specific cron-apt configuration that I use is summarised in one action file: /etc/cron-apt/action.d/5-install

dist-upgrade -y -o APT::Get::Show-Upgraded=true -o Dir::Etc::SourceList=/etc/apt/sources.list.d/security.list -o Dir::Etc::SourceParts="/dev/null"

Any other upgrade, is done manually, using screen or whatever is most appropriate, as it may require manual intervention during the upgrade.

share|improve this answer
add comment

you can use Fabric. Fabric is a Python (2.5-2.7) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.

share|improve this answer
add comment

For years I've been happily upgrading and installing packages using apt-dater. It is lightweight and effective tool for remote package management. It uses screen, sudo and ssh.
For package management apt-dater may be easier solution than configuration management tools.
apt-dater is handy for centralised package management on different GNU/Linux flavours such as Debian and CentOS.

share|improve this answer
add comment

The answer using exec is pretty helpful.

However according to the apt-get manual it's not a good idea to use -q=2 this way (though I have used it for years without problems)

-q, --quiet
       Quiet; produces output suitable for logging, omitting progress indicators. More q's will produce more quiet up to a maximum of 2. You can also use -q=# to set the
       quiet level, overriding the configuration file. Note that quiet level 2 implies -y, you should never use -qq without a no-action modifier such as -d, --print-uris or
       -s as APT may decided to do something you did not expect. Configuration Item: quiet.

I have used a script myself for years, running apt-get the following way:

ssh example.org "apt-get update && apt-get -y upgrade && apt-get -y dist-upgrade && apt-get clean"

Things like puppet and other tools people mentioned sure may work, but it seems like it's overkill for what basically is just mimicking a few commands typed by a human. I believe in using the simplest tool for a specific job, in this case a bash script is about as simple as it gets without losing functionality.

share|improve this answer
    
Yes, I think its a good way for some Servers. But if I should get some more options (rollout some configs on each server, etc.) then its puppet the really better way. I work at the moment on puppet and its beautiful.. –  Dennis Wisnia Jan 7 '12 at 10:30
    
I don't disagree. But for the purpose of what the poster was asking about it may be overkill. –  aseq Jan 10 '12 at 22:08
add comment

if all your hosts are debian, you can try the unattended-upgrades package.

http://packages.debian.org/sid/unattended-upgrades

Here we have been using puppet to manage our debian virtual machines, with puppet we are able to enable and manage unnatended-upgrade configs on all servers.

Recently our team are testing the mcollective tool to run commands on all servers, but to use mcollective ruby skills are needed.

[s] Guto

share|improve this answer
add comment

I would recommend going for Puppet, facter and mCollective.

mCollective is a very nice framework where you can run commands over a series of hosts (in parallels) using facter as filter.

Add to that a local proxy / cache and you'd be well set for servers management.

share|improve this answer
    
I agree, Puppet isn't really the best tool to manage administrative actions like mass/orchestrated package updates. –  robbyt Jan 3 '12 at 19:44
add comment

So I guess there are many things which contribute to a good solution:

  • Bandwidth
  • Ease of administration
  • Detailed logging in case something screws up.

Bandwidth: Basically two alternatives to save bandwidth come into my mind:

  • Setting up a Debian mirror and configure all your clients to use this mirror, see http://www.debian.org/mirror/ for more details. (I would recommend this)
  • Setting up a proxy (apt-cacher, apt-proxy or Squid) and increase cache so all your clients can profit from this cache

Administration: I would configure a parallel shell like PDSH,PSSH,GNU Parallel and issue the command on all clients, if I tested the command previously on an example machine. Then its not very likely that it may fail on all the others. Alternatively you may consider a cron job on all clients, but then it may fail automatically, so I would prefer the first solution.

If you concern about simultaneity of upgrades you could schedule your commands with at

Logging: As with parallel shells you have the possibility to redirect output I would combine stderr and stdout and write it to a logfile.

share|improve this answer
    
Bandwidth: there are caching proxies specific for deb repositories, look for apt-cacher or apt-proxy. –  S19N Jan 3 '12 at 9:39
    
Great I will integrate this in the answer. –  math Jan 3 '12 at 11:51
    
and software such as mCollective will allows parallel commands execution AND report the output/outcome. –  CloudWeavers Jan 3 '12 at 19:52
add comment

My own parallel ssh wrapper: classh is an alternative to the various Parallel and cluster ssh tools out there.

You might like it better or you might hate it. There are only three reasons I'm mentioning it here:

  • It's extremely simple to install and use: a single .py file with no external dependencies beyond the Python 2.5 standard libraries.
  • It's extremely reliable within its limits. I use it every working day, often nearly 100 times per day and usually on collections of hundreds to a few thousand targets per command. (I've tested it on target lists of well over 25 thousand servers at a time). It's never failed to run, failed to complete or given me any indeterminate behavior. (The only limitations related to those of the Python subprocess.communicate() method --- so you can only get capture about 64K of stdout and, separately up to 64K of stderr, for example; also any remote process which attempts to read from its stdin will simply stall until the local ssh subporcess is killed, automatically by classh's timeout handling)
  • It's extremely simple to write a custom script, in Python, to use classh.py as a module. So it's very easy to write something like:

    
        !#/bin/env python
        import classh
        job = classh.SSHJobMan(cmd, targets)
        job.start()
        while not job.done():
            completed = job.poll()
            for i in completed:
                # do something with the classh.JobRecord object referenced by i
        # done

    # You can optionally do post-processing on the dictionary of JobRecords here # keyed off the target strings (hostnames) </code></pre>

That's all there is to it. For example in the nested completed loop you can gather a list of all those which returned some particular exit status or to scan for specific error messages, and set up follow-up jobs to handle those. (The jobs will be run concurrently, default of 100 jobs at any time, until each is completed; so a simple command on a few hundred hosts usually completes in a few seconds and a very complex shell script in a single long command string ... say fifty lines or so ... can complete over a few thousand hosts in about 10 minutes ... about 10K hosts per hour in my environment, with many of those located intercontinentally).

So this might be something you can use as an ad hoc measure until you have your puppet configuration implemented and well testing ... and it's also quite handing for performing little ad hoc surveys of your hosts to see which ones are deviating from your standards in various little ways.

share|improve this answer
    
Incidentally on the classh web pages, at bitbucket.org there is also a list of the various other ssh wrappers that I surveyed before I decided to write my own. Any of them might work for you. Additionally you might look up Python Fabric which is a newer project with similar, though somewhat more extensive and a bit more complex, features. –  Jim Dennis Jan 3 '12 at 11:35
add comment

Use a tool that is made to run a single command on multiple servers. And by that I do not mean having a kazillion terminals open with Terminator or ClusterSSH, but instead having a single terminal to a management server running a tool suitable for the job.

I would recommend func, Salt or mCollective in this context. If you already have Puppet, go for mCollective (it integrates nicely in Puppet). If you don't, and you have an old Python on your machines, you might enjoy func. If you Python in new, try Salt. All these tools run the command specified at the command line asynchronously, which is a lot more fun than a sequential ssh loop or even doing the same aptitude commands in umpteen Terminator windows to umpteen servers.

You'll definitely love Salt.

share|improve this answer
add comment

use webmin,,,and use its webmin cluster feature, in which you can add all systems to one webmin console and issue them any command or control all of them from one place.

Or

Use Cluster ssh

Or

PSSH

share|improve this answer
    
Yes, I really can open a lot of windows and can work in it parallel. Cluster SSH is pretty cool but I think its not smart enough. –  Dennis Wisnia Jan 3 '12 at 7:44
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.