How do you gracefully upgrade mission critical systems to wildly disparate systems?

Question

In the span of the 12+ years of my career, I have yet to overcome this hurdle and I suspect the answer simply isn't easy or even possible, so I ask everyone here for their experience.

Say that you're running into egregious problems that can only be fixed by moving from one platform to another - either from making a mistake in choosing the platform that was chosen years ago, or simply growing beyond what the system was originally designed for. You know for certain that the cruft that has built up over time will invariably mean that it will be nearly impossible to test for all the things that will certainly lead to tech support hell - which we all know leads to the loss of customers. Not that customers aren't already complaining about the egregious problems that already exist!

The best possible way that I've discovered so far is to maybe devise a plan for the changeover, test it on a few clients, test it on a dozen clients, test it on a hundred clients, then finally finish the changeover for everyone and pray that you've worked out all the bugs with those first hundred and twenty, and that the animal by-products will not hit the ventilation system in the most spectacular fashion possible.

However, that doesn't mean that it won't anyway.

So say that you're moving from Exchange to Exim (or even just Sendmail to Exim). How do you handle it?

Answer 1

That is a tricky question and varies based on the specific situation. The key issues are:

• have a fallback plan What ever you do, have a plan for how to go back to the old systems quickly. If you cannot do this, then more testing is required.
• testing your most common functionality. Look at your logs or measure which features are used most and are more critical. Test that functionality more thoroughly.
• Have several users, including yourself "live" on the new server for multiple days. The length of time should depend on how drastic the change. (i.e. "you should be the first tester)
• run the new service in parallel with the old, if possible. If you can put a proxy in the middle and forward certain requests or users, that can help.
• run in a load-balanced cluster. This is similar to the previous statement. If you can run both services in a load-balanced arrangement, try that. You can gradually get rid of the old service as things go smoothly.
• Keep the old servers around for a few weeks or more. I have kept old servers around for months when I wasn't sure if I would have to use them. Turn them off or disconnect the network cable to ensure that there are no hidden dependencies on the server.
• Make sure the new service can handle the load. Deploy it gradually and monitor the system performance while gradually moving more traffic to the new system.
• when doing things, make sure failures are visible and immediate. you want failures to occur early and be easily identified.

Use DNS to your advantage. Set up both services to respond to the same DNS name, but have DNS point to one or the other, (or both as a round-robin). Use the local hosts file on Linux and Windows to override DNS and be able to verify the setup before rollout and after. This also makes for easier troubleshooting after rollout. Just change the local hosts file to the old server and see if things are still broken on the problem machine. Set you TTL's low to allow for quick fallback. Load balancers like Cisco's GSS can be used for this. I can use iptables to take a specific load-balanced host out of the pool.

For Apache, using a reverse proxy is a good way to migrate a site piecemeal. For others, Use DNS, a proxy, or possibly an iptables box to give you options on how to control the transition.