7

I basically have the same problem as mentioned in this thread -- I would like to temporarily suspend all requests to all servers of a certain backend, so that I can upgrade the backend and the database it uses. Since this is a live system, I would like to queue up requests, and send them to the backend servers once they've been upgraded. Since I'm doing a database upgrade with the code change, I have to upgrade all backend servers simultaneously, so I can't just bring one down at a time.

I tried using the tcp-request options combined with removing the static healthcheck file as mentioned in that thread, but had no luck. Setting the default "maxconn" value to 0 seems to pause and queue connections as desired, but then there seems to be no way to increase the value back to a positive number without restarting HAProxy, which kills all requests that had been queued up until that point. (The "hot-reconfiguration" options using -sf and -st start a new process, which doesn't seem to do what I want).

Is what I'm trying to do possible?

Improve this question
1
  • Do you have some health-check?
    – Thomas Berger
    Oct 13, 2012 at 19:56

3 Answers 3

Reset to default
11

I eventually ended up asking this question of Willy Tarreau, the author of HAProxy. He was intrigued by my suggestion, and committed a small change to HAProxy that allows for setting maxconn down to zero via the admin socket (this wasn't possible at the time I asked), which solved my problem. Quoting from the followup email I sent him:

Hi there. That solves my problem pretty well. I issued "set maxconn frontend my_frontend 0", waited a few seconds for connections to drain, and then all subsequent connections are paused. I restarted the server, issued "set maxconn frontend my_frontend 3000", and connections resumed properly, without erroring out on existing requests.

In response to JesseP's answer -- absolutely, most of the time I never want to do this. We generally try and stage our DB migrations in exactly the way you mention, because suspending traffic is risky at best. Some of our users set ridiculously low client-side timeouts, so we generally don't want traffic suspended for more than 15 seconds. But for a recent migration where we had a complex set of code and data migrations to perform simultaneously, having this option available was a lifesaver.

So, to sum up -- not recommending this for everyday use, but the option is there should the need arise.

Improve this answer
4
  • Blog post about this technique here: blog.balancedpayments.com/…
    – Marc
    Dec 23, 2014 at 19:02
  • 1
    Do you know if setting maxconn to 0 for a frontend disconnects long-living connections, like websockets and/or HTTP keep-alive connections?
    – Saurabh Nanda
    Feb 20, 2019 at 6:21
  • 1
    It's been years since I wrote this, but I believe the answer is "no" -- that's why I wrote that it was necessary to wait "a few seconds for connections to drain." If you have long timeouts on the client side, this technique may not work for you.
    – Marc
    Mar 7, 2019 at 18:40
  • There is a maxconn setting for server, which would allow us to queue all connections to the application without waiting for clients to drop. However "0" for maxconn server means unlimited, not 0...
    – remram
    Jan 3, 2022 at 17:06
1

Even if you accomplish what you're asking, you're going to end up with longer running data migrations, too many servers to update, etc., in the future, and clients will end up timing out while they wait for stuff to come back online.

You should really try to build your updates such that existing servers can still run. This may involve deploying some updated code with existing database, then updating the database, then deploying the actual update.

In the end, you should be able to deploy one server at a time, with out breaking things.

Improve this answer
1
  • What do you suggest if we have to move alive production database from one datacenter to another?
    – CMCDragonkai
    Aug 15, 2014 at 10:18
1

I couldn't manage to use haproxy's approach using set maxconn frontend myfrontend 0 followed by set maxconn frontend myfrontend 1000 because my haproxy crashed when I perform the second command to unblock the requests.

I took an alternative approach with OpenResty which I documented here: https://github.com/svilen-ivanov/antract

antract is an OpenResty set of scripts written in Lua to help you perform simple zero downtime maintenance on the backend service. Its goal is to limit the impact to the user by holding incoming requests "in flight" while the backend service is taken down for maintenance and started back up again. The user sees a long request, and things carry right along.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .