11

We have many servers and still want to update them all. The actual way is that any of the sysadmins go from server to server and make a aptitude update && aptitude upgrade - it's still not cool.

I am searching now for a solution which is still better and very smart. Can puppet do this job? How do you do it?

1
  • yes, puppet can do this. cssh would fix your problem in the short term, also.
    – Sirex
    Jan 3, 2012 at 7:43

11 Answers 11

10

You can use the exec type such as:

exec { "upgrade_packages":
    command => "apt-get upgrade -q=2",
    path    => "/usr/local/bin/:/bin/:/usr/bin/",
    # path  => [ "/usr/local/bin/", "/bin/" ],  # alternative syntax
}

To be honest, I did not try it myself, but I think you just need to create a new module that include such an exec definition.

The apt-get upgrade command is interactive. To make it run quietly, you can add the option -q=2 as shown above.

6
  • looks very good! I think I create an Puppet Usecase with some Testmachines to try this. Many Thanks! Jan 3, 2012 at 8:34
  • 3
    +1 for recommending Puppet ! Changes your life as a sysadmin :) Jan 3, 2012 at 10:27
  • 3
    Mind that this exec will run on every Puppet run (every 30 minutes) which might hammer your proxy and / or your mirror pretty hard if you have a "lot of servers". Personally, I would recommend implementing a schedule for the exec type above, making sure it only runs at night, for example. In my opinion though, Puppet is meant to enforce system state, and running an upgrade_packages-like command without human oversight through it, is both a bit scary and a bit of Puppet abuse. The mColective tool that comes with Puppet Enterprise (or it's open source equivalent) might be a better option.
    – wzzrd
    Jan 3, 2012 at 10:31
  • 7
    Our puppet installation has a similar exec that checks the timestamp on a file and only runs the upgrade if it's newer than the version on the client. When we want to upgrade everything, we touch that file on the puppet master.
    – Ladadadada
    Jan 3, 2012 at 10:44
  • @wzzrd: Good point, but it can be made better by checking some external condition as Ladadadada said.
    – Khaled
    Jan 3, 2012 at 10:57
7

if all your hosts are debian, you can try the unattended-upgrades package.

http://packages.debian.org/sid/unattended-upgrades

Here we have been using puppet to manage our debian virtual machines, with puppet we are able to enable and manage unnatended-upgrade configs on all servers.

Recently our team are testing the mcollective tool to run commands on all servers, but to use mcollective ruby skills are needed.

[s] Guto

5

I would recommend going for Puppet, facter and mCollective.

mCollective is a very nice framework where you can run commands over a series of hosts (in parallels) using facter as filter.

Add to that a local proxy / cache and you'd be well set for servers management.

1
  • I agree, Puppet isn't really the best tool to manage administrative actions like mass/orchestrated package updates.
    – robbyt
    Jan 3, 2012 at 19:44
3

Use a tool that is made to run a single command on multiple servers. And by that I do not mean having a kazillion terminals open with Terminator or ClusterSSH, but instead having a single terminal to a management server running a tool suitable for the job.

I would recommend func, Salt or mCollective in this context. If you already have Puppet, go for mCollective (it integrates nicely in Puppet). If you don't, and you have an old Python on your machines, you might enjoy func. If you Python in new, try Salt. All these tools run the command specified at the command line asynchronously, which is a lot more fun than a sequential ssh loop or even doing the same aptitude commands in umpteen Terminator windows to umpteen servers.

You'll definitely love Salt.

2

So I guess there are many things which contribute to a good solution:

  • Bandwidth
  • Ease of administration
  • Detailed logging in case something screws up.

Bandwidth: Basically two alternatives to save bandwidth come into my mind:

  • Setting up a Debian mirror and configure all your clients to use this mirror, see http://www.debian.org/mirror/ for more details. (I would recommend this)
  • Setting up a proxy (apt-cacher, apt-proxy or Squid) and increase cache so all your clients can profit from this cache

Administration: I would configure a parallel shell like PDSH,PSSH,GNU Parallel and issue the command on all clients, if I tested the command previously on an example machine. Then its not very likely that it may fail on all the others. Alternatively you may consider a cron job on all clients, but then it may fail automatically, so I would prefer the first solution.

If you concern about simultaneity of upgrades you could schedule your commands with at

Logging: As with parallel shells you have the possibility to redirect output I would combine stderr and stdout and write it to a logfile.

3
  • Bandwidth: there are caching proxies specific for deb repositories, look for apt-cacher or apt-proxy.
    – S19N
    Jan 3, 2012 at 9:39
  • Great I will integrate this in the answer.
    – math
    Jan 3, 2012 at 11:51
  • and software such as mCollective will allows parallel commands execution AND report the output/outcome. Jan 3, 2012 at 19:52
1

My own parallel ssh wrapper: classh is an alternative to the various Parallel and cluster ssh tools out there.

You might like it better or you might hate it. There are only three reasons I'm mentioning it here:

  • It's extremely simple to install and use: a single .py file with no external dependencies beyond the Python 2.5 standard libraries.
  • It's extremely reliable within its limits. I use it every working day, often nearly 100 times per day and usually on collections of hundreds to a few thousand targets per command. (I've tested it on target lists of well over 25 thousand servers at a time). It's never failed to run, failed to complete or given me any indeterminate behavior. (The only limitations related to those of the Python subprocess.communicate() method --- so you can only get capture about 64K of stdout and, separately up to 64K of stderr, for example; also any remote process which attempts to read from its stdin will simply stall until the local ssh subporcess is killed, automatically by classh's timeout handling)
  • It's extremely simple to write a custom script, in Python, to use classh.py as a module. So it's very easy to write something like:

    
        !#/bin/env python
        import classh
        job = classh.SSHJobMan(cmd, targets)
        job.start()
        while not job.done():
            completed = job.poll()
            for i in completed:
                # do something with the classh.JobRecord object referenced by i
        # done

    # You can optionally do post-processing on the dictionary of JobRecords here # keyed off the target strings (hostnames) </code></pre>

That's all there is to it. For example in the nested completed loop you can gather a list of all those which returned some particular exit status or to scan for specific error messages, and set up follow-up jobs to handle those. (The jobs will be run concurrently, default of 100 jobs at any time, until each is completed; so a simple command on a few hundred hosts usually completes in a few seconds and a very complex shell script in a single long command string ... say fifty lines or so ... can complete over a few thousand hosts in about 10 minutes ... about 10K hosts per hour in my environment, with many of those located intercontinentally).

So this might be something you can use as an ad hoc measure until you have your puppet configuration implemented and well testing ... and it's also quite handing for performing little ad hoc surveys of your hosts to see which ones are deviating from your standards in various little ways.

1
  • Incidentally on the classh web pages, at bitbucket.org there is also a list of the various other ssh wrappers that I surveyed before I decided to write my own. Any of them might work for you. Additionally you might look up Python Fabric which is a newer project with similar, though somewhat more extensive and a bit more complex, features.
    – Jim Dennis
    Jan 3, 2012 at 11:35
1

The answer using exec is pretty helpful.

However according to the apt-get manual it's not a good idea to use -q=2 this way (though I have used it for years without problems)

-q, --quiet
       Quiet; produces output suitable for logging, omitting progress indicators. More q's will produce more quiet up to a maximum of 2. You can also use -q=# to set the
       quiet level, overriding the configuration file. Note that quiet level 2 implies -y, you should never use -qq without a no-action modifier such as -d, --print-uris or
       -s as APT may decided to do something you did not expect. Configuration Item: quiet.

I have used a script myself for years, running apt-get the following way:

ssh example.org "apt-get update && apt-get -y upgrade && apt-get -y dist-upgrade && apt-get clean"

Things like puppet and other tools people mentioned sure may work, but it seems like it's overkill for what basically is just mimicking a few commands typed by a human. I believe in using the simplest tool for a specific job, in this case a bash script is about as simple as it gets without losing functionality.

2
  • Yes, I think its a good way for some Servers. But if I should get some more options (rollout some configs on each server, etc.) then its puppet the really better way. I work at the moment on puppet and its beautiful.. Jan 7, 2012 at 10:30
  • I don't disagree. But for the purpose of what the poster was asking about it may be overkill.
    – aseq
    Jan 10, 2012 at 22:08
1

For years I've been happily upgrading and installing packages using apt-dater. It is lightweight and effective tool for remote package management. It uses screen, sudo and ssh.
For package management apt-dater may be easier solution than configuration management tools.
apt-dater is handy for centralised package management on different GNU/Linux flavours such as Debian and CentOS.

1

you can use Fabric. Fabric is a Python (2.5-2.7) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.

0

use webmin,,,and use its webmin cluster feature, in which you can add all systems to one webmin console and issue them any command or control all of them from one place.

Or

Use Cluster ssh

Or

PSSH

1
  • Yes, I really can open a lot of windows and can work in it parallel. Cluster SSH is pretty cool but I think its not smart enough. Jan 3, 2012 at 7:44
0

Another solution if all your hosts are running Debian (or derivatives) is to use the cron-apt package. But, as suggested per the documentation, a bit of care must be taken.

I'm currently using cron-apt on a dozen of servers to perform all the security updates automatically and unattended. To avoid any unwanted upgrades, I only use cron-apt on servers which runs the Debian stable distribution and I make sure to configure my apt sources so use the distribution name, wheezy, and not its alias (stable).

The specific cron-apt configuration that I use is summarised in one action file: /etc/cron-apt/action.d/5-install

dist-upgrade -y -o APT::Get::Show-Upgraded=true -o Dir::Etc::SourceList=/etc/apt/sources.list.d/security.list -o Dir::Etc::SourceParts="/dev/null"

Any other upgrade, is done manually, using screen or whatever is most appropriate, as it may require manual intervention during the upgrade.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .