I have three Tomcat instances running behind an HAProxy. When I roll out changes to my web applications, I'd like to do a rolling deploy (i.e. bounce one Tomcat at a time) so users don't see any downtime.

How do I do this? I see that a running instance of haproxy can be hot-reconfigured (which is good for adding or removing new pool servers), but how does HAProxy react when one of its target servers is temporarily down?

If there's a better solution than HAProxy, I'd be willing to look at that too.



I sent an e-mail to Willy Tarreau and got back the following options:

  1. You can use hot reconfiguration or socat commands to the stats server to set the weight of your target server to 0. This will prevent new sessions from being balanced to that server, but won't affect existing connections.

  2. You can set the http-check disable-on-404 option in combination with "option httpcheck /myurl" and then have your target servers respond to /myurl such that they send a 200 status if everything is fine, a 404 if the server should stop receiving new requests, and a 500 if the server should receive nothing (i.e. when you're ready to bounce the server). haproxy will re-check on the server at the interval specified in your server line.

|improve this answer|||||

Being as that HAProxy does not support removal without reconfiguration according to @Ernest Muller in his answer, I will provide you an alternative as you requested other scenarios as well.

I use LVS, which is one of my favorite solutions for load balancing, as it can be used for more than HTTP.

With LVS, you can use ipvsadm to manually add and remove servers. An example of removal is the following command:

/sbin/ipvsadm -e -t VIP:443 -r SERVERIP:443 -g -w WEIGHT

As opposed to manually adding and removing by interacting with LVS, I often prefer to use the request option with ldirectord. ldirector is a daemon that polls and manages your ipvs configuration. You can specify a file with a URI path using that parameter. During rollout, you remove the file and wait for the server to stop being accessed. At that point, you can deploy the code without impacting production clients.

|improve this answer|||||

A lot of this depends on whether you are doing any session/state management on the Tomcats. If a user's session is destroyed by the restart, then rolling doesn't prevent user impact (it might prevent them from seeing a 500, but not from having to start their session over). If you're not using sticky sessions than you probably don't have to worry about that.

HAproxy, and other load balancers, have ways to try to pretty quickly determine if the server behind it is up or down and reroute traffic based on that (the "health check," in HAProxy). It is impossible for them to do that perfectly, however. With Tomcat, there isn't just "up" and "down"; there's "up, as in responding on the port, but things aren't ready yet". So you shouldn't rely on the LB to completely prevent user facing impact - even with a good health check there will be an interval where you're getting traffic to a bad node.

What we do with a rolling deploy is to actively take the server out of the load balancer, then mess with/restart the node, wait till it passes an automated test/monitor, then put it back in, then move to the next server. This is easier with a load balancer that has an API you can call remotely (like, from a script) to disable a server - our old Netscaler did this, but HAProxy doesn't. With HAProxy you either have to edit config and restart (sad) or you can change the health check to where you can manipulate it - like maybe it checks a magic file you rename when you want it to omit that node. You have to wait for the health check to fire and the node to go out of the cluster, but then you should be fine.

I ran across this post that has a iptables-related solution to that problem...

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.