Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I maintain 2 environments in my current project, 2 servers (1 Web Server & 1 SQL Server) for both production and Test. Last month we installed/upgraded to the lated Microsoft patches/securities and the Report Manager from Reporting Services stopped working in the Test database server. After days of troubleshooting I found that the ASP.NET 2.0 Web Service Extension in ISIS was completely removed and everything was set to prohibited.

I can't say for sure but I think this was caused by the patch/updated that I'd done on that server a few day prior. What can one do to anticipate or prevent these types of impacts on SQL Server, Reporting Services, IIS, ASP.NET when a patch is installed?

share|improve this question

3 Answers 3

(coming from a linux point of view - but pretty generic)

  1. Don't let the OS automatically install patches - just download them and notify you. Then you can install them manually so that if something goes wrong you can address it right away
  2. I configure all of my critical servers in clusters, so that they can be patched one at a time, and rebooted without any downtime - and if something goes wrong, it buys me some grace time to fix it, and plan better for the next one.
  3. I always apply the patch first to a less-critical server so that if it ruins everything, nobody will notice :)
  4. Our environment is virtual, which allows us to snapshot the server state before doing anything to it (such as patching). Perhaps the equivalent would be taking a full backup of the host before patching.
  5. have a rollback plan that will allow you to get back to the state the system was in before making any changes.
share|improve this answer

One approach is evaluate the patches and don't just install them right away. We have a group of people who specifically review patches for Microsoft and other common applications, rate the severity of a vulnerability and talk to the application owners about risks, etc.

Some patches are deemed critical and get applied within 48 hours (or less) of release. We began patching the RPC vulnerability last fall within 30 minutes of its release. Others we'll wait up to 6 weeks depending on the severity, the ability to mitigate the risk of exploitation, and test effort required.

That way you have time to let other people get burned, and you avoid applying the initial rev of a patch. It's not a perfect process, and may be a little to intense for many folks, but it works for us.

share|improve this answer

You should continue to keep a test environment that mimics your production environment as closely as possible and to keep current documentation of your configurations. Then when patch loads are released continue to test them in your test environment. If something starts dying then verify the configuration after the patch update(s) with your current documented configuration to see if anything / what was changed. Then try to replicate that change via the patch update again if necessary to pinpoint it as the culprit.

As far as anticipating and preventing them, that's a hard one to come to grips with. Err on the side of paranoia and don't install anything in production until you've fully tested it and are comfortable that it is working as advertised. The majority of the time all will be well. Make sure that your configuration is fully supported so that you can call support in the cases that something bombs.

When applying to production, always have a tested rollback plan so that you can get back to original state when something whacks out. (that's a technical term)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.