2

I'm asking this question of the SF community based on the latest posting from Dreamhost (a significant webhosting provider here in the US) based on significant downtime suffered from them:

We run Debian OS and have used autoupdates to ensure security packages are installed as soon as they are available. We’ve had some breakage in the past from this approach, but nothing major. However last night’s autoupdate went badly wrong, removing essential packages from dedicated, VPS and some shared servers.

I'm curious because my approach has always been to never autoupdate my Windows Servers, test patches and track installed patches diligently. I even go so far as to let a couple of days pass after patch Tuesdays to see if anyone else is experiencing issues with a just released patch. Since I don't maintain *nix servers as a general rule, I was wondering if it's different for *nix?

Am I too conservative? What do you do?

Edited to add: Here is more info from their site

Our monitoring and support team flagged the issue fast, and we scrambled our admin, dev and NOC teams to reinstall the packages that had been removed by autoupdate, reboot servers, fix package dependencies, and test that individual services were live. Given the number of services affected, this took a long time to complete. Rest assured we had all hands working on the issue, but I know it was still a frustrating experience for customers.

To mitigate the risk of anything like this happening again, we’re immediately switching off autoupdates, and moving to a manual process where we’ll only push out Debian updates after significant testing.

2
  • This is absolutely not a technical question, but an invitation to debate. Jan 30, 2012 at 17:31
  • 1
    There is a lot of room for opinion, but I'm going to say that patch management is ultimately a calculable science, and @Bart has already provided a great answer demonstrating that. Jan 30, 2012 at 18:59

3 Answers 3

7

It is something dependent on your site. With proper backups, if something goes drastically wrong you can roll back changes by restoring to the previous update.

Take a risk assessment. Automatically doing anything risks having something automatically break. Can you stand to have your server unavailable for the time it takes to restore it in the worst case scenario?

If you can't be down at all, don't autoupdate.

If you can't be down for the time it takes to do a restore, don't autoupdate.

If you can afford the risk, autoupdate so you get the latest updates as soon as possible.

Or do it regularly by hand.

Generally *nix updates won't break everything, but generally Windows updates don't break everything either. It's the edge cases you worry about. The dependencies. Evaluate it on a case by case basis and assess the risk you want to take. And make sure your backups are recent.

7
  • 1
    It just seems to me, that autoupdating the amount of servers that DH must run, is asking for trouble. Let's just say for the sake of argument that they only have 100 servers. Having 100 servers go down at once because they autoupdated seems, at best, ridiculous and at worst, well, it might put you out of business.
    – GregD
    Jan 30, 2012 at 16:40
  • They might not ordinarily reboot the whole farm at once. Ideally they'd have a testbed and local repo they roll out changes from. Jan 30, 2012 at 16:43
  • 2
    And automate their updates with Chef or Puppet, something like that. Jan 30, 2012 at 16:43
  • There's autoupdate from Windows (pulling from a central site and reboot automatically) and there's autoupdating with automation tools, separate thing. You wouldn't want to manage 100's of servers without some kind of automation to manage updates. But you don't want random updates rolled to your systems either. Jan 30, 2012 at 16:44
  • From the additional comments that I added to my OP, it doesn't sound like they were using any automation to manage updates..which adds to my confusion as to why you'd do this
    – GregD
    Jan 30, 2012 at 16:49
2

The dreamhost debacle is self commenting actually. I'd say unsupervised updates are even worse on *nix then on *doze, because appication packages get updated as well (on top of strictly system packages). You do not want a major MySql, Postgres or php upgrade to go past you unaware.

My .02, obviously.

1
  • For those who do not know, Fedora (among other distros) had a history of pushing postgresql updates that required a DB dump/restore. They fixed it (maybe) lately, but that is just one example... Jan 30, 2012 at 16:52
2

Wait and see also isn't necessarily the conservative approach that you think it is either. For security patches that close exploits, that time when the patch is first released is when you are most likely to be hit by an automated attack. Like a lot of things it is about finding a balance.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .