Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am using EC2 as the live production server, and I am using git for source control. Currently, when I push changes to the site, I have a bash script that runs & pulls the most recent changes from git, and restarts the server. This causes the site to be down for about ten seconds while this is happening.

I would like to be able to push changes and keep the site live. (My initial thought would be to fire up an additional EC2 instance with the existing code while these changes are being pushed to the server that is being updated.) What are some ways to accomplish keeping the site live while pushing changes?

share|improve this question
    
Why do you need to restart anything when pushing code changes to a site? Is it like compiled and running in memory or? –  Frands Hansen Jan 2 '12 at 4:37
    
It is a python/django project, and for all the changes to take place you need to restart the server. –  David542 Jan 2 '12 at 5:17
add comment

2 Answers 2

up vote 4 down vote accepted

I like to do a transition from a server running old code to a server running new code using a load balancer.

If you have enough live servers that you can handle twice the current traffic, you can:

  1. Take half of your servers out of service
  2. Upgrade those servers.
  3. Switch the servers running old code out of service an the servers running new code into service (atomically).
  4. Upgrade the second half.
  5. Bring all servers into service.

If you don't have enough running servers to spare, you can follow a procedure like this:

  1. Bring up new servers with the new code.
  2. Add the new servers to the load balancer and bring them into service while taking the old servers out of service.
  3. Throw away the old servers or leave them on standby for a fast rollback if necessary.

This works best for stateless instances like web servers. A database requires a different approach, but often your web servers are using a single database that does not need to change that often, and your database changes can be made to work with both the old and new code for the transition.

We have found one issue when using Amazon's Elastic Load Balancer (ELB) to do the above type of live switchover to new servers. When you take the old instances out of service, the ELB rudely drops any active connnections. Depending on your application you may be able to live with it, especially if your previous behavior was a complete outage with a server restart.

Here's the thread at Amazon where we reported this bug. You can follow it for updates:

Existing connections dropped rudely when instance taken out of ELB
https://forums.aws.amazon.com/thread.jspa?threadID=61278

Jump in with your +1 if this is an issue for you as Amazon has asked for further feedback there.

share|improve this answer
add comment

You probably want the "graceful restart" rather than a normal "restart". In Apache, you can probably do /etc/init.d/apache2 graceful (Ubuntu) or /etc/init.d/apache graceful (RHEL/CentOS) to cause new connections to use the newly updated instance of the application. Another benefit is that existing requests won't be interrupted while the restart is done.

From your description, this should solve your problem. See the "graceful" section of man apachectl for more details about what "graceful" means.

share|improve this answer
    
It's worth a shot to see if this works for you, but Apache's graceful restart has never been sufficient for my production systems. Part of the issue for me is that I like to have the parent process load up code and often used data so that it is in shared memory when the children are forked off. –  Eric Hammond Jan 2 '12 at 9:04
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.