We're setting up HAProxy to balance our application (ASP.NET MVC 3 running on IIS). We want our process to be without scheduled maintenance for deployment. I'm trying to figure out the correct way to do a "see-saw" approach to only serve one version of the application to new requests at a single time. Here's what I've come up with so far:

  1. Remove the first half of the servers from HAProxy by reloading the config:

    $ sed -i 's/web01.*/& disabled/' /etc/haproxy/haproxy.cfg
    $ sed -i 's/web02.*/& disabled/' /etc/haproxy/haproxy.cfg
    $ /etc/init.d/haproxy reload
  2. Update the application on each down instance and poke them with curl to warm them.

  3. Bring back the first half and remove the second half:

    $ sed -i 's/\(web01.*\) disabled$/\1/' /etc/haproxy/haproxy.cfg
    $ sed -i 's/\(web02.*\) disabled$/\1/' /etc/haproxy/haproxy.cfg
    $ sed -i 's/web03.*/& disabled/' /etc/haproxy/haproxy.cfg
    $ sed -i 's/web04.*/& disabled/' /etc/haproxy/haproxy.cfg
    $ /etc/init.d/haproxy reload
  4. Repeat step 2 for the second half.

  5. Bring back the second half:

    $ sed -i 's/\(web03.*\) disabled$/\1/' /etc/haproxy/haproxy.cfg
    $ sed -i 's/\(web04.*\) disabled$/\1/' /etc/haproxy/haproxy.cfg
    $ /etc/init.d/haproxy reload

With the approach described above, the existing requests would gracefully continue using V1 of the app and then V2 for any subsequent requests. I'm OK with this because it'd significantly reduce the chance of serving the two different versions.

However, using sed to replace the config seems like a hack. Here are my other alternatives I've thought of using:

  1. Use the UNIX socket to do this same "see-saw" approach. However, because the UNIX socket can only work with one server at a time and the deployment could take a few seconds per-server, it could lead to HTML at V2 but JavaScript for V1. If anything, it's much easier to write backwards compatible code than forwards compatible. So using this approach seems like it might not work.

  2. Use rolling deployments across the servers with either UNIX sockets or the health check. During my research, I've come across this approach as being most popular. However, this leads to the same problems as the UNIX sockets - having to write forward compatible code.

Maybe I'm completely over complicating this or missing something obvious...

  • Out of curiosity what are doing at the DB tier to achieve your zero downtime deployment? – HTTP500 Feb 12 '13 at 16:30
  • @HTTP500 - We try to minimize migrations that require locking tables etc. So, for example, we do a column drop over two migrations (since requests to V1 still need the column). Otherwise, we'd have to schedule some sort of maintenance or put the app into "readonly" mode (which we have yet to implement). Check out this slide deck that describes what I mean (it's RoR but still applies). – TheCloudlessSky Feb 12 '13 at 16:39
  • We may or may not go full blown app compatible... but it's something we have to consider. – TheCloudlessSky Feb 12 '13 at 16:39

So, you're worried about some assets not lining up version-wise with each other. This isn't really a load-balancer problem but a code problem. A common practice is to 'version' the assets using a parameter. E.g., my.html might contain:

<link rel="stylesheet" href="http://foo.com/css/blah.css?v123">

You should have a global setting somewhere in your code which you increment on release. In any case, you need to force a reload of the assets client-side as they'll be cached there.

| improve this answer | |
  • Right - and this is what we currently do and don't have an issue with. However, there are other requests that aren't versioned (such as AJAX requests). With a rolling-restart, you could serve HTML V2 of your app, and then do an AJAX request to a WS that is only V1. – TheCloudlessSky Feb 12 '13 at 15:50
  • Ah, I see. That's tricky alright. The only thing I can think of is having a cookie with the version number in it. HAProxy would need to check this and redirect to the appropriate backend. If the cookie is set => go to server with that version, if cookie not set => set to new version. Your code would need to be forward compatible to some degree here - you would need to increment this version setting on the old servers when you do a deployment so that new sessions always go to new servers. Plus, it could take a long time to deploy - you'd need to wait for the old sessions to taper off. – chrskly Feb 13 '13 at 10:33
  • Yeah - I've thought about that too. Having the forward-compatible requirement seems a little silly though. I'm really curious as to how others do this type of deployment. Every article that I've read pretty much uses rolling deployments... but they don't really talk about any of the issues I've brought up. – TheCloudlessSky Feb 13 '13 at 12:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.