14

I just RDP'd into one of my companies servers, was alerted to windows updates, so I click. Then I see 62 high priority updates, with the last update (according to update history) was installed on Thursday, January 16, 2014, more than one year ago.

What actions need to be taken here?

12
  • 21
    Consider yourself fortunate that mfinni and others are actually answering this. It is akin to one of us coming to SO and asking "when I write code should I debug it?"
    – TheCleaner
    Feb 9, 2015 at 15:21
  • 7
    @TheCleaner The answer to that question is "after you upsell the customer on your code-debugging services." Feb 9, 2015 at 15:28
  • 8
    @MonkeyZeus "if it ain't broke..." in this case, you mean "if it ain't secure, don't secure it" ?
    – user186340
    Feb 9, 2015 at 17:15
  • 5
    "If it ain't broke, don't fix it" and "If it ain't secure, don't secure it" express essentially opposite ideas. Feb 10, 2015 at 1:06
  • 7
    @Lilienthal - "useful for many other developers" has no bearing on this site. This site isn't designed as a helpdesk for SO users. Call it cruel if you want, I didn't make the site's scope.
    – TheCleaner
    Feb 10, 2015 at 14:29

4 Answers 4

31

Short answer - yes. Most of the Windows Updates are security related. Not having the patches means you're vulnerable.

Longer answer - you need a procedure that covers this sort of thing. It's more rare these days, but sometimes a patch can break things, or change behavior in such a way that it's broken as far as your company is concerned. You should be evaluating each patch when it's released (there's a monthly schedule plus some urgent ones), determine if you need the patch (probably yes), do some testing on test/staging servers to do some diligence about potential breakage, and then do the installs.

You should also exercise some care about the deployments, because OS patching often means rebooting, which often means there's service downtime, unless you've got some good HA for all your services. If you think you'll be clever and patch during the day and then postpone the reboot, that's not a great idea - some files will be updated but others won't.

Microsoft offers a free product called WSUS that can make patch management a little easier than doing approvals and deployment all one-by-one.

FYI, you should be doing this sort of thing for all classes of device you have. Network device firmware, server hardware firmware, VMware ESXi, etc. Those patches don't come out for the fun of it, almost all of them address bugs, and many of them can be security related.

Further - you should be asking someone who's more senior than you on your technical team. If you're the only admin there, you and your organization are not doing too well. Don't take that personally, we all need to start without knowing everything we should - but if this is your question, you shouldn't be the only person managing these servers.

7
  • 14
    Fast-typing bastard. >:/ Feb 9, 2015 at 15:00
  • 1
    Snow day baby. Trying to get VPN access into the office.
    – mfinni
    Feb 9, 2015 at 15:03
  • I'm not managing them, I'm an app developer who happened to need to view some event viewer logs on the host, I've actually noticed update alerts before but this time I missed the little 'x' and clicked the bubble, leading me to the summary page. My dilemma now is what sort of flag do I raise to senior management, because it appears to me that the work is simply not being done. We actually have WSUS. Up until today I just assumed that any update notice I saw would be taken care of that weekend.
    – OpenCoderX
    Feb 9, 2015 at 15:41
  • Talk to management immediately. Do you have systems administrators? If you do, then they might not be doing their job, unless your company policy is "don't install updates." If you don't have systems administrators, get management to hire some or contract it out. As you can probably guess, devs don't have the same goals or skillsets as sysadmins and most can't/shouldn't play both roles.
    – mfinni
    Feb 9, 2015 at 15:44
  • 10
    "My dilemma now is what sort of flag do I raise to senior management, because it appears to me that the work is simply not being done. " - no dilemma, you tell your boss via email what you noticed and are concerned. There may be a legit reason, or it may simply be laziness. Either way, it isn't your fault it's not been done, but you should at least voice concern.
    – TheCleaner
    Feb 9, 2015 at 15:44
18

The generic answer is it is a good practice to keep your servers updated.

But pay attention to a few things:

  1. Updates may cause the server to be sluggish during installation, or even cause some downtime if they require reboot(s). You should plan to do them out of office work hours.

  2. Updates have some risk associated. They might break your server, or cause some incompatibility. They are usually fully uninstallable, but with 62 of them you should also consider if you have a trustworthy backup (you should, anyway).

  3. Is there a reason why you are one year late on upgrades? Is this your first log in to that server in a year, or is something else broken?

  4. Pay special attention to the infamous Excel bug that comes with some December updates of Office, if your company uses Excel macros, but this probably doesn't apply to a server that shouldn't be running Office.

  5. Many sysadmins wait a few days or weeks before installing updates, just to see if anything bad comes up on the Internet regarding those updates. When deciding if you need to wait, consider security risks of leaving the server unpatched for more time.

7
  • What "infamous Excel bug that comes with some December updates of Office" are you talking about?
    – nobody
    Feb 11, 2015 at 0:44
  • "For some users, Form Controls (FM20.dll) are no longer working as expected after installing MS14-082 Microsoft Office Security Updates for December 2014." as per Technet blog post blogs.technet.com/b/the_microsoft_excel_support_team_blog/…
    – Shiv
    Feb 11, 2015 at 4:36
  • @Shiv: thanks, I edited the answer to include your link.
    – pgr
    Feb 11, 2015 at 10:12
  • @pgr, Aren't there like tons of these infamous bugs?
    – Pacerier
    Feb 12, 2015 at 10:04
  • @Pacerier: eheh, sure. Usually all you have to do is rollback the update. Not this one. Files can get "infected" with the bug, i.e., somebody opens them after the bad update, and suddenly the file stops working on a different computer. It's been a real PITA dealing with this one, and it's not over yet. Notice that the issue has got so complex (for the worst cases, when the problem travels with the file) that Microsoft is STILL working on it, and a definitive solution is still to be achieved... but, of course, each sysadmin will have his own nightmare story, this is mine... :-)
    – pgr
    Feb 12, 2015 at 12:09
8

I know mfinni beat me to the punch, but I'm just going to +1 for WSUS. Specifically:

Let's assume that you have multiple servers, including test and production. Let's also assume that test has similar hardware to production (which isn't a safe assumption, I know, but let's go with it--it's nice but not necessary). You could set up the following scenario in WSUS:

  1. Test servers in their own OU. Group policy says to install updates and reboot at some non-inconvenient time, like Sunday at 3am.
  2. Prod servers in a different OU or OUs. Group policy says to download and notify.
  3. Patches approved, and deadlined to install and reboot the servers during your scheduled maintenance window, several days or a week after the test/dev servers apply patches.

What this does, if it's not obvious, is it approves all critical/security patches for your servers, applies them to test first, and then applies them later to production. I've only seen an update critically break something once, but this would give you a chance to roll back the patch if it fails in test before it applies to prod.

As for the big pile of updates on the server in question, patching is a lower risk than not patching, but I would verify my backups before applying them all just in case because there are so many. If it's a VM, you might want to take a snapshot first.

1

This is entirely up to your business and the policy that you have set out for updating your servers.

At the very least you should install security updates and perform any other patches like .NET framework updates in a testing environment first before updating production servers.

12
  • 2
    1. Too slow. You got beat to the punch by two other, better answers. 2. There's nothing opinion-based about whether to install patches/security updates or not. The only scenario I can envision where you wouldn't want to install patches would be one where you're stealing from your employer. 3. "Patch management" is most definitely a Server Fault topic, though it might also be topical at super User. Feb 9, 2015 at 15:05
  • 1
    If I knew my server admins were asking this on SF I'd be terrified for my infrastructure. The core of the question is "What should I do?" not something akin to "How do I manage/automate/improve?" which would fall under the category of patch management and so on. I thought this place was for professionals, maybe I am wrong about that. Just seems like it belongs on SU to me! Feb 9, 2015 at 15:13
  • 1
    The asker is clearly fairly junior, because he/she is asking this question. They need help; that's why this site exists. Both other answers are "Yes, here's more details and nuance."
    – mfinni
    Feb 9, 2015 at 15:18
  • 5
    I would be more worried about the server admins who did not ask and did not update for a year. Feb 9, 2015 at 15:41
  • 3
    It's definitely something that should be raised; there have been some fairly critical security updates in the last 12 months. Feb 9, 2015 at 15:48

Not the answer you're looking for? Browse other questions tagged .