I like to do a transition from a server running old code to a server running new code using a load balancer.
If you have enough live servers that you can handle twice the current traffic, you can:
- Take half of your servers out of service
- Upgrade those servers.
- Switch the servers running old code out of service an the servers running new code into service (atomically).
- Upgrade the second half.
- Bring all servers into service.
If you don't have enough running servers to spare, you can follow a procedure like this:
- Bring up new servers with the new code.
- Add the new servers to the load balancer and bring them into service while taking the old servers out of service.
- Throw away the old servers or leave them on standby for a fast rollback if necessary.
This works best for stateless instances like web servers. A database requires a different approach, but often your web servers are using a single database that does not need to change that often, and your database changes can be made to work with both the old and new code for the transition.
We have found one issue when using Amazon's Elastic Load Balancer (ELB) to do the above type of live switchover to new servers. When you take the old instances out of service, the ELB rudely drops any active connnections. Depending on your application you may be able to live with it, especially if your previous behavior was a complete outage with a server restart.
Here's the thread at Amazon where we reported this bug. You can follow it for updates:
Existing connections dropped rudely when instance taken out of ELB
Jump in with your +1 if this is an issue for you as Amazon has asked for further feedback there.