Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Currently we have the following setup.

  • Hardware Load Balancer (LB)
  • Box A running Tomcat on 8080 (TA)
  • Box B running Tomcat on 8080 (TB)

TA and TB are running behind LB.

For now it's pretty complicated and manual job to take Box A or Box B out of LB to do the zero downtime deployment.

I am thinking to do something like this:

  • Hardware Load Balancer (LB)
  • Box A running Nginx on 8080 (NA)
  • Box A running Tomcat on 8081 (TA1)
  • Box A running Tomcat on 8082 (TA2)
  • Box B running Nginx on 8080 (NB)
  • Box B running Tomcat on 8081 (TB1)
  • Box B running Tomcat on 8082 (TB2)

Basically LB will be directing traffic between NA and NB now. On each of Nginx's we'll have TA1, TA2 and TB1, TB2 configured as upstream servers. Once one of the upstreams's healthcheck page is unresponsive (shutdown) the traffic goes to another one (HttpHealthcheckModule module on Nginx).

So the deploy process is simple.

Say, TA1 is active with version 0.1 of the app. Healthcheck on TA1 is OK. We start TA2 with Healthcheck on it as ERROR. So Nginx is not talking to it. We deploy app version 0.2 to TA2. Make sure it works. Now, we switch the Healthcheck on TA2 to OK, switch Healthcheck to TA1 to ERROR. Nginx will start serving TA2, and will remove TA1 out of rotation. Done! And now same with the other box.

While it sounds all cool and nice, how do we "starve" the Nginx? Say we have pending connections, some users on TA1. If we just turn it off, sessions will break (we have cookie-based sessions). Not good. Any way to starve traffic to one of the upstream servers with Nginx?

Thanks!

share|improve this question
1  
Why is telling the load balancer to drain connections such a big deal? This is something it should be able to handle easily. –  longneck Oct 9 '12 at 2:25

2 Answers 2

up vote 3 down vote accepted

My opinion is if you're willing to put in such a complicated setup just to get zero-downtime, you should instead be investing your time in getting a better load balancer. By going down the path you suggest, you are trading simplicity for marginal cost savings, which always ends up costing you more than you anticipate.

You're going to need downtime to implement this zero-downtime solution you propose. Take that downtime to instead figure out how to use your existing load balancer, or swap it out for something better.

share|improve this answer

I accomplish basically what your looking for by using HAProxy as my load balancer and having it do frequent checks against a "health check" html page. In my nginx config on the backend web servers, this page doesn't physically exist. During deployment, I create a file on the backend server called "offline.html". In nginx, if this file exists when "check.html" is requested, it serves it, otherwise it serves the real health check page.

During my deployments, I touch the offline.html file, wait a few seconds for the loadbalancer to kick out the server, deploy, then remove the file.

Site fires back up, load balancer adds the server back into the pool, and everything works perfectly...

Side note, I'd try to remove reliance on a session affinity if you can...

share|improve this answer
    
We have similar way. We just manually redirects the traffic to nodeA while nodeB is going to be upgraded, then after a sanity check on nodeB, we switch all traffic to nodeB and upgrade nodeA. Do not need another balancer layer (nginx). –  ruffp Oct 25 '13 at 10:07

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.