I am using HAProxy socket communication to put my backend into maintenance mode before bringing the web server down for maintenance via this command:

echo "disable server cluster-01/app-01-4" | sudo socat stdio /tmp/haproxy.sock

The problem is that when I mark the backend available again via the socket enable command, HAProxy does not wait for any successful health checks before sending requests to the backend. So we end up losing some of the requests before HAProxy realizes the server is still down, and takes it back off line.

Is there a way to tell HAProxy to wait until the backend is healthy prior to enabling it? Or do I have to make my startup scripts smart enough to monitor the server itself prior to telling HaProxy to enable the backend again? This would obviously be less than ideal.

I'm currently doing this:
echo "enable server cluster-01/app-01-4" | sudo socat stdio /tmp/haproxy.sock

Is there something akin to this that I can use instead? echo "enable server cluster-01/app-01-4 WAIT_FOR_SUCCESSFUL_HEALTH_CHECK" | sudo socat stdio /tmp/haproxy.sock

Edit: I am using HAProxy version 1.5x

  • What version of HAProxy? – Zypher Mar 3 '16 at 22:48
  • @Zypher Version 1.5.14 – WickedElephant Mar 4 '16 at 1:35
  • Is it possible to go to the 1.6.x line? – Zypher Mar 4 '16 at 1:49
  • @Zypher Yes, if it will get me what I am looking for. What does 1.6.x provide that will solve this? – WickedElephant Mar 4 '16 at 2:02
  • What about set server cluster-01/app-01-4 state ready? (After disabling it with ...state maint) or set server cluster-01/app-01-4 health down before enable server...? – Michael - sqlbot Mar 4 '16 at 4:08

Using set server proxy-name/server-name health down prior to putting the server back in service with enable server ... appears to do what you need.

The server would eventually be taken out of service by subsequent health checks, but that doesn't help if the server was healthy when you took it out, but not healthy when you bring it back in, since HAProxy appears to assume that the server will be in the same health state when enabled that it was in when it was disabled.

| improve this answer | |
  • Thanks again, this was exactly what I was looking for and works perfectly. – WickedElephant Mar 7 '16 at 18:10

You can move to haproxy 1.6.3 and use the new agent-check functionality. This also lets you drain hosts out as you take them out of rotation.

| improve this answer | |
  • Thank you very much for your help Zypher. It does seem like 1.6.3 provides the functionality I seek out of the box. The word "up" sets back the server's operating state as UP if health checks also report that the service is accessible. In the end, I am going with Michael - sqlbot's answer because it works with HAProxy 1.5 as a workaround. – WickedElephant Mar 4 '16 at 18:53

At Loadbalancer.org we do this is a slightly complicated way :-).

Before we restart HAProxy we check all of the current states and check for any servers that are DOWN or MAINT.

Then we BLOCK SYN packets (handles the dropped traffic on reload issue) Then we restart HAProxy

Then we manually set all the states to DOWN or MAINT (based on what they were 0.01 seconds ago..)

Then we re-enable SYN packets...

Giving us the required result of seamless restarts with no downtime or lost packets.

If you can't be bothered with ALL the above then just set everything to DOWN state after a reload ..do it fast and cross your fingers :-).

v1.6 has some new state-full restart code but I haven't researched it yet.

| improve this answer | |
  • Thanks for your reply, but it sounds like you are talking about restarting HAProxy, which is not what my question is about. I am talking about doing rolling restarts of my application/web servers, and how to keep HAProxy from sending requests to them until they are healthy once I re-enable them in HAProxy. – WickedElephant Mar 4 '16 at 18:40
  • Sorry , I misunderstood however it should still a similar process: Like the guy said previously put the server in MAINT state. Do the maintenance and then set the server to READY/DOWN state... and as soon as you bring the server back online and it passes a health check it will change to UP state. Or use the feedback agent in file monitor mode: loadbalancer.org/blog/… – Malcolm turnbull Mar 6 '16 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.