Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
  • When upgrading or installing a package through a package manager like Aptitude, what do you do to ensure everything works afterwards?

  • If the package manager would break something after running, do you have a checklist or plan to recover from it?

  • Are there any ways to minimize the risk of something breaking when updating a package manager, and if so: how do you do it?

share|improve this question
add comment

3 Answers

up vote 3 down vote accepted

Ensuring everything works after upgrading

  • Scrutinize the changes to the package list before upgrading. Many problems I've seen in this department were people blindly trusting the tools and then failing to notice that apache went away because of some other slip-up.

  • Check the output of apt while upgrading. This can give immediate feedback if something fails to start again.

  • Have a proper and in-depth monitoring solution configured. Nagios would be a start.

Emergency plans

  • Have a current backup ready.

  • Know what you're doing. Don't press buttons just because they flash.

  • Switch over to the hot-standby server.

Minimizing risk

  • Use only packages from your distribution.

  • Use only packages from the same release.

  • Don't install non-packaged software.

  • Don't install foreign packages.

  • Try the upgrade first on your QA server.

share|improve this answer
add comment

My experience with with rpm and yum/yumex, but it translates.

For a critical package being upgrade, I just make normal use of the package to ensure that it's working correctly. If I upgrade bind, then I do some DNS requests that I expect to succeed. If I upgrade Apache httpd then I open some web pages. If I upgrade open-office then I open a document or two.

On those rare occasions when a package upgrade leaves things fubar, my procedure to recover is:

  1. Uninstall the failed package (sometimes I have to "force" the uninstall)
  2. Manually remove any temporary or configuration files left after the uninstall
  3. Re-install the upgraded package that failed
  4. Test

If after step #4 above it fails again, then I repeat but fall back to the previous version.

Before any major upgrade, I'll ensure that my backup is up-to-date. This saved my butt once when I did an upgrade only to find out that the new version wasn't yet compatible with my video card, and was documented as such. (My failure to fully read the release notes. Shame on me.) It also saved me once when during an upgrade my hard disk failed.

Thus, to answer your 3rd question about minimizing risk:

  • Back up!
  • Always read release notes
  • If you're not the first to upgrade, and you are nervous about it, check the internet to see if others are complaining about problems
  • Have a recovery disk ready -- for most modern Linux distributions, the install disk is also a recovery disk. Find out what recovery mechanism your distribution provides. If possible, boot via CDRom into recovery mode so you understand how it works, so you understand how to chroot into your installation from recovery mode.
share|improve this answer
add comment

Monitoring and testing. Monitoring makes sure it's still working (whether the package manager craps out or not), and if you're doing a major upgrade, you test things beforehand. Virtualisation can help you easily duplicate a production environment, automation can help you build a production replica, and having a permanent staging environment gives you somewhere to try things out.

When you're doing an upgrade, or any non-trivial maintenance, you should have the entire process completely scripted, including rollback and post-upgrade testing.

"The Practice of System and Network Administration" is a great book for systems administrators, describing these sorts of processes and how to go about putting together checklists and such.

share|improve this answer
    
+1 Hi Matt, well met! :) –  David Schmitt May 4 '09 at 10:04
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.