up vote 2 down vote favorite
2
share [g+] share [fb]

we are in the process of optimizing our network.

Maybe you heard that while deploying SP1 for W7, some computers using a WSUS powered network had problems, and the computers crashed/had to be redone.

Now, there are several ways to prevent these kinds of desasters. Usually, you have some kind of spare machine with a mirroring installation that you try the updates on first. If it comes back up without a problem, you patch the original machines and hope it works out just as well.

But what is the best way to do just that? IT infrastructure is complex, and even if I had a test machine with the same OS as the live machine, the update could go through while the live machine will crash. Because the complex software installation does not match.

So I could say that I create a test machine with the same OS, OS patch level and applications, but it could still crash. As noted earlier, the latest crash victims depended on the use of a WSUS server in the domain. A computer not using the WSUS wouldn't have gotten the same problem.

So, the solution for this would be to clone every live machine when updates wait for installations, boot the clone, then update the clone. But that gets us into more trouble, specifically with Microsoft and their Windows licenses, as I then have 2 machines running the same license, contacting Windows Update. Not to speak about the other doubles licenses running on the machine.

Is there a clever solution for the problems listed here? How do you prevent update desasters on your live servers without having a physically mirrored machine?

link|improve this question
75% accept rate
1  
Easy: never update anything. Windows NT 4.0 forever! Wooooo! – womble Jul 11 '11 at 13:27
feedback

2 Answers

up vote 2 down vote accepted

Backup your stuff and do the update.

Prior to patching, read the release notes and browse around online for any known issues. If you are worried about it, do the upgrade on a test machine that has the same software installed as your machines in use. This obviously doesn't guarantee that you won't have a problem, but is generally considered close enough.

Worst case scenario, the patch breaks something. For many Windows updates, you can simply uninstall them. Others, you can use System Restore to go back. If the whole OS is hosed (never seen it happen), then simply re-install Windows in place. If that doesn't work, and you cannot repair it, then simply start over with a formatted hard drive, install windows, and restore your data.

To my knowledge, there is no special trick, only best practices. Despite the hype, updates are generally not that scary.

link|improve this answer
We'll generate a process of patch notes reading and googling before patching any servers. Meanwhile, I will keep a close eye on Full Disclosure. Thanks Brad. – Dabu Jul 12 '11 at 6:26
Brad, can you give me a time estimate for actually finding problems with the patches? Should I google around on patch day, 1, 2 days after? There are updates today and I try to find information about problems with the patches, but do not find any. I cannot currently distinguish between "there are no problems" and "it's too early to tell if there are problems". – Dabu Jul 13 '11 at 5:23
@Dabu, it depends on the patches. Some problems don't crop up until years later. Others are immediate. You have to weigh that risk with the risk of not patching. When you patch depends on your individual needs. – Brad Jul 13 '11 at 13:15
feedback
up vote 0 down vote

We sit a quarter behind to make sure of no outstanding issues. Prior to doing the updates, we'll research for any known issues and make note of them before approving in change management. Then we'll do a pilot group before system wide deployment. If no issues in pilot after a week, we'll roll out wide.

link|improve this answer
2  
good luck with those being-exploited-in-the-wild-right-now vulnerabilities. – Michael Lowman Jul 11 '11 at 13:53
1  
So your mean-time-to-patch is about 6 months?! – Hubert Kario Jul 11 '11 at 14:02
Security-wise, this is a problem. Public knowledge suggest that about 90% of all attacks on security holes are done in the week after the release of the patch, maybe starting with the post on Full Disclosure and the likes. Servers with external exposure should usually be patches ASAP. – Dabu Jul 11 '11 at 14:03
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.