Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm running an HAProxy load balancing server to balance load to multiple Apache servers. I need to reload HAProxy at any given time in order to change the load balancing algorithm.

This all works fine, except for the fact that I have to reload the server without losing a single packet (at the moment a reload is giving me 99.76% success on average, with 1000 requests per second for 5 seconds). I have done many hours of research about this, and have found the following command for "gracefully reloading" the HAProxy server:

haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/ -sf $(cat /var/run/

However, this has little or no effect versus the plain old service haproxy reload, it's still dropping 0.24% on average.

Is there any way of reloading the HAProxy config file without a single dropped packet from any user?

share|improve this question
If you need that much reliability a better solution would be to run more than one instance of HAproxy where you can take one out of service to reload, put it back in and repeat for the other(s). – yoonix Mar 7 '14 at 21:19
up vote 18 down vote accepted

According to and consequently you can:

iptables -I INPUT -p tcp --dport $PORT --syn -j DROP
sleep 1
service haproxy restart
iptables -D INPUT -p tcp --dport $PORT --syn -j DROP

This has the effect of dropping the SYN before a restart, so that clients will resend this SYN until it reaches the new process.

share|improve this answer
both of these commands gave me this: iptables v1.4.14: invalid port/service --syn' specified` – Dmitri DB Oct 28 '14 at 21:26
@DmitriDB you're supposed to replace $PORT with the actual port haproxy is listening on. If haproxy is listening on multiple ports, write replace --dport $PORT with --dports $PORTS_SEPARATED_BY_COMMAS, e.g., --dports 80,443. – pepoluan Dec 18 '14 at 11:17
iptables 1.4.7 (Centos 6.7) - you have to also specify -m mulitport if you want to use --dports. So its "iptables -I INPUT -p tcp -m multiport --dports 80,443 --syn -j DROP" and likewise for the -D – carpii Sep 16 '15 at 21:47

Yelp shared a more sophisticated approach based on meticulous testing. The blog article is a deep dive, and well worth the time investment to fully appreciate it.

True Zero Downtime HAProxy Reloads

tl;dr use Linux tc (traffic control) and iptables to temporarily queue SYN packets while HAProxy is reloading and has two pids attached to the same port (SO_REUSEPORT).

I'm not comfortable re-publishing the entire article on ServerFault; nevertheless, here are a few excerpts to pique your interest:

By delaying SYN packets coming into our HAProxy load balancers that run on each machine, we are able to minimally impact traffic during HAProxy reloads, which allows us to add, remove, and change service backends within our SOA without fear of significantly impacting user traffic.

nl-qdisc-add --dev=lo --parent=1:4 --id=40: --update plug --buffer
service haproxy reload
nl-qdisc-add --dev=lo --parent=1:4 --id=40: --update plug --release-indefinite

iptables -t mangle -I OUTPUT -p tcp -s --syn -j MARK --set-mark 1

## Set up the queuing discipline
tc qdisc add dev lo root handle 1: prio bands 4
tc qdisc add dev lo parent 1:1 handle 10: pfifo limit 1000
tc qdisc add dev lo parent 1:2 handle 20: pfifo limit 1000
tc qdisc add dev lo parent 1:3 handle 30: pfifo limit 1000

## Create a plug qdisc with 1 meg of buffer
nl-qdisc-add --dev=lo --parent=1:4 --id=40: plug --limit 1048576
## Release the plug
nl-qdisc-add --dev=lo --parent=1:4 --id=40: --update plug --release-indefinite

## Set up the filter, any packet marked with “1” will be
## directed to the plug
tc filter add dev lo protocol ip parent 1:0 prio 1 handle 1 fw classid 1:4


Cheers to Yelp for sharing such amazing insights.

share|improve this answer
Excellent link! But perhaps you'd like to summarize it in here in case the link expires. That's the only reason for no upvote. – Matt May 20 '15 at 22:54
@Matt added some excerpts and code samples – Steve Jansen Jun 1 '15 at 20:00

If you're on a kernel that supports SO_REUSEPORT, then this problem shouldn't happen.

The process that haproxy takes when it restarts is:

1) Try setting SO_REUSEPORT when opening the port (

2) Try opening the port (will succeed with SO_REUSEPORT)

3) If it didn't succeed, signal the old process to close its port, wait 10ms and try it all again. (

It was first supported in the Linux 3.9 kernel but some distros have backported it. For example, EL6 kernels from 2.6.32-417.el6 support it.

share|improve this answer
It will happen with SO_REUSEPORT under some particular scenario - especially under heavy traffic. When SYN is sent to old haproxy process and in the same moment it closes listening socket which results in RST. See Yelp article mentioned in other answer above. – gertas Dec 30 '15 at 23:43
That sucks... Just to summarise the issue, Linux distributes new connections between all processes listening on a particular port when SO_REUSEPORT is used so there is a short time where the old process will still get connections put into its queue. – Jason Stubbs Jan 4 at 11:19

There is another much simpler way to reload haproxy with true zero downtime - it is named iptables flipping (the article is actually Unbounce response to Yelp solution). It is cleaner than accepted answer as there is no need to drop any packets which may cause problems with long reloads.

Briefly, the solution consists of the following steps:

  1. Let's have a pair of haproxy instances - the first active which receives a traffic and the second in standby which does not receive any traffic.
  2. You reconfigure (reload) standby instance at any time.
  3. Once standby is ready with new config you divert all NEW connections to standby node which becomes new active. Unbounce provides bash script which does the flip with few simple iptable commands.
  4. For a moment you have two active instances. You need to wait till opened connections to old active will cease. The time depends on your service behaviour and keep-alive settings.
  5. Traffic to old active stops which becomes new standby - you are back in step 1.

Moreover the solution can be adopted to any kind of service (nginx, apache etc) and is more fault tolerant as you can test standby configuration before it goes online.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.