How Often Should I Update our Linux Server?

Question

I am responsible for managing both our production server (mail, web, database are all on one server) and our test server. Both are built on Debian. However as I am very new to system administration, I have only been installing updates as I come across things that have to be updated so that I can have newer features and get bug fixes. Its a pretty ad hoc process right now, and I'd like to make it less so.

So I am wondering how people who know what they're doing handle this? How often do you perform upgrades on your servers? Is the upgrade process different between test and production? Do you always upgrade any test servers first? And do you do a full update of all software, or do you just install selected updates?

Answer 1

I run apt-get update -qq; apt-get upgrade -duyq daily. This will check for updates, but not do them automatically.

Then I can run the upgrades manually while I am watching, and can correct anything that might go wrong.

Besides the security concerns of maintaining a patched system, I find that if I leave it too long between patches, I end up with a whole bunch of packages that want to be upgraded, and that scares me a-lot more than just upgrading one or two every week or so. Therefore I tend to run my upgrades weekly, or if they are high priority, daily. This has the added advantage of knowing which package broke your system (ie. if you're only upgrading a couple at a time)

I always upgrade less critical systems first. I also have a "rollback plan" in place in case I can't fix the system. (since most of our servers are virtual, this rollback plan usually consists of taking a snapshot before the upgrade that I can revert to if necessary)

That being said, I think an upgrade has broken something only once or twice in the past 4 years, and that was on a highly customized system - so you don't have to be TOO paranoid :)

Answer 2

On top of previous answers - a couple more specifically Debian things: you should Subscribe to debian-security-announce and debian-announce and / or check out the Debian Security page.

Answer 3

Assuming you are running the stable release of Debian, most patches will be security or bug related which should mean that there won't be too many major changes between versions of any given package. According to the debian patch policy patches should also have been in testing for some time before being moved to the stable branch by the maintainer. Obviously This won't stop breakages when patching but should prevent them in most cases.

It would be prudent to ensure that your testing sever is kept upto date and any packages that have bugs affecting you and your servers should be kept up to date. All packages that have security advisories against them should be updated as soon as you know the patch is stable.

Debian is usually a very stable OS and not one you should be overly concerned with breakages on however always read what is going to be updated before it get updated and keep an eye out for anything that seems strange. I use VCS on my /etc/ dir as well to ensure that any config file changes can be seen with a 'git diff' command.

Answer 4

Where I work we have a pretty extensive process that involves using software called PatchLink to notify us of the most important security related updates, and we apply them after testing, on a package by package basis. We have thousands of servers though.

If you only have two servers the process should be much simpler. Although I do not think doing an "apt-get update/upgrade" is your best bet.

I would monitor patches for the software you are running, and make decisions based on the fixes in those releases on when to upgrade.

Since you have a test server, obviously, always test the update before applying them.

Answer 5

On debian i install cron-apt and edit its configuration file to mail me if ther are any changes. this way i get notified if there are updates for my systems and do the updates by hand

Answer 6

I do a dry run (first) to see what is going to be updated. Sometimes, libraries (lets call it libfoo for this example) change their API, which breaks programs that we wrote / installed ourselves. If some critical library is updated, I grab the source and try rebuilding our stuff against it prior to updating.

I also check to see that we're not jumping to an intermediate version of some public service, ie apache, etc. I'd rather stay a year behind and not encounter random breakage, unless the update is critical.

If you are a system administrator, you should be pulling RSS feeds from sites like Secunia, which should let you know way ahead of time if your distro is going to be pushing some patches.

Do not ever, ever just blindly upgrade / update. Unfortunately, the task of knowing what is broken falls on you, not your distro package manager, especially if your systems support programmers.

Answer 7

I like cron-apt to automate this process, but as @dinomite pointed out on another question regarding updates, configuring it specifically to automate security-related updates is a very smart idea -- you can then manually update what you need. I had been using cron-apt for all updates but actually changed this based on his answer. If you like it, you should probably vote his answer up rather than this one.

Answer 8

Along the same lines as cron-apt you should take a look at the unattended-upgrades package http://packages.debian.org/lenny/unattended-upgrades.

Its very easy to configure and will enable you to have security updates downloaded and applied automatically but leave other updates for manual upgrading (or at your discretion upgrade everything!).

The Official Ubuntu Server Guide, has a reasonably detailed section covering the use of the unattended-upgrades package https://help.ubuntu.com/9.04/serverguide/C/automatic-updates.html

Note: depending on your level of caution/paranoia you can do a rolling upgrade on a group of test servers first, then if there are not issues, allow your production boxes to update, although I personally have not come across any trouble with security updates wrecking havoc so far (knock on wood)...

There is also a config option to mail you the results of each security update as well, once its applied. Also, if there were any dialog or interactive prompts that were presented during the update, ones that will need manual tweaking by a sysadmin, it will mention these as well.

Answer 9

Besides the stuff that is mentioned you should use some kind of monitoring tool (Nagios or whatever floats your boat) to alert you of updates.

As far as how often goes: As soon as there's an update available!