Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am responsible for managing both our production server (mail, web, database are all on one server) and our test server. Both are built on Debian. However as I am very new to system administration, I have only been installing updates as I come across things that have to be updated so that I can have newer features and get bug fixes. Its a pretty ad hoc process right now, and I'd like to make it less so.

So I am wondering how people who know what they're doing handle this? How often do you perform upgrades on your servers? Is the upgrade process different between test and production? Do you always upgrade any test servers first? And do you do a full update of all software, or do you just install selected updates?

share|improve this question

9 Answers 9

up vote 21 down vote accepted

I run apt-get update -qq; apt-get upgrade -duyq daily. This will check for updates, but not do them automatically.

Then I can run the upgrades manually while I am watching, and can correct anything that might go wrong.

Besides the security concerns of maintaining a patched system, I find that if I leave it too long between patches, I end up with a whole bunch of packages that want to be upgraded, and that scares me a-lot more than just upgrading one or two every week or so. Therefore I tend to run my upgrades weekly, or if they are high priority, daily. This has the added advantage of knowing which package broke your system (ie. if you're only upgrading a couple at a time)

I always upgrade less critical systems first. I also have a "rollback plan" in place in case I can't fix the system. (since most of our servers are virtual, this rollback plan usually consists of taking a snapshot before the upgrade that I can revert to if necessary)

That being said, I think an upgrade has broken something only once or twice in the past 4 years, and that was on a highly customized system - so you don't have to be TOO paranoid :)

share|improve this answer
2  
I work very hard to touch each server every 30 days. I have 80+ servers at this point. I do them in batches by functional group or by operating system. –  Thomas Denton May 18 '09 at 17:13
1  
We have a cron script that runs the equivalent for our SLES/OpenSuSE boxes nightly; when it finds that it needs packages, it submits a ticket to the system administration queue in our trouble ticket system. (It keeps track of which ones it's submitted before in a file in /tmp so that it doesn't spam the queue.) –  Karl Katzke Jun 6 '09 at 22:23
3  
Debian has two packages, apticron and cron-apt, which do a similar thing and email you if there are any updates available. IME, apticron has the edge by emailing you the changelog so you can see what has changed. –  David Pashley Jun 6 '09 at 22:28

On top of previous answers - a couple more specifically Debian things: you should Subscribe to debian-security-announce and debian-announce and / or check out the Debian Security page.

share|improve this answer

Assuming you are running the stable release of Debian, most patches will be security or bug related which should mean that there won't be too many major changes between versions of any given package. According to the debian patch policy patches should also have been in testing for some time before being moved to the stable branch by the maintainer. Obviously This won't stop breakages when patching but should prevent them in most cases.

It would be prudent to ensure that your testing sever is kept upto date and any packages that have bugs affecting you and your servers should be kept up to date. All packages that have security advisories against them should be updated as soon as you know the patch is stable.

Debian is usually a very stable OS and not one you should be overly concerned with breakages on however always read what is going to be updated before it get updated and keep an eye out for anything that seems strange. I use VCS on my /etc/ dir as well to ensure that any config file changes can be seen with a 'git diff' command.

share|improve this answer

Where I work we have a pretty extensive process that involves using software called PatchLink to notify us of the most important security related updates, and we apply them after testing, on a package by package basis. We have thousands of servers though.

If you only have two servers the process should be much simpler. Although I do not think doing an "apt-get update/upgrade" is your best bet.

I would monitor patches for the software you are running, and make decisions based on the fixes in those releases on when to upgrade.

Since you have a test server, obviously, always test the update before applying them.

share|improve this answer

On debian i install cron-apt and edit its configuration file to mail me if ther are any changes. this way i get notified if there are updates for my systems and do the updates by hand

share|improve this answer

I do a dry run (first) to see what is going to be updated. Sometimes, libraries (lets call it libfoo for this example) change their API, which breaks programs that we wrote / installed ourselves. If some critical library is updated, I grab the source and try rebuilding our stuff against it prior to updating.

I also check to see that we're not jumping to an intermediate version of some public service, ie apache, etc. I'd rather stay a year behind and not encounter random breakage, unless the update is critical.

If you are a system administrator, you should be pulling RSS feeds from sites like Secunia, which should let you know way ahead of time if your distro is going to be pushing some patches.

Do not ever, ever just blindly upgrade / update. Unfortunately, the task of knowing what is broken falls on you, not your distro package manager, especially if your systems support programmers.

share|improve this answer

I like cron-apt to automate this process, but as @dinomite pointed out on another question regarding updates, configuring it specifically to automate security-related updates is a very smart idea -- you can then manually update what you need. I had been using cron-apt for all updates but actually changed this based on his answer. If you like it, you should probably vote his answer up rather than this one.

share|improve this answer

Along the same lines as cron-apt you should take a look at the unattended-upgrades package http://packages.debian.org/lenny/unattended-upgrades.

Its very easy to configure and will enable you to have security updates downloaded and applied automatically but leave other updates for manual upgrading (or at your discretion upgrade everything!).

The Official Ubuntu Server Guide, has a reasonably detailed section covering the use of the unattended-upgrades package https://help.ubuntu.com/9.04/serverguide/C/automatic-updates.html

Note: depending on your level of caution/paranoia you can do a rolling upgrade on a group of test servers first, then if there are not issues, allow your production boxes to update, although I personally have not come across any trouble with security updates wrecking havoc so far (knock on wood)...

There is also a config option to mail you the results of each security update as well, once its applied. Also, if there were any dialog or interactive prompts that were presented during the update, ones that will need manual tweaking by a sysadmin, it will mention these as well.

share|improve this answer

Besides the stuff that is mentioned you should use some kind of monitoring tool (Nagios or whatever floats your boat) to alert you of updates.

As far as how often goes: As soon as there's an update available!

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.