3

We're using HAProxy 1.6.3 to load balance and route HTTP traffic to hundreds of backend servers. We reload the config often (several times a day), both automatically when a server fails and manually for administrative reasons.

The problem is that running the reload command takes up to 3 minutes on one of our HAProxy servers (Ubuntu 16.04). It does not seem to matter whether the server has traffic or not. On our other servers with the same version of OS and HAProxy, the reload takes 1-5 seconds, regardless of load. We have a bunch of long-running requests but as I said it does not seem to matter if the server has traffic or not.

We can see that a new process is spawned, but then it takes a few minutes before it starts accepting traffic (or at least until the CPU usage of the new process climbs above 0%).

The question is: What can cause HAProxy to take so long to reload? What is it doing for so long? How can I find out (for instance, what level of logging do I need to enable and what would I look for in the log?)

We run the following command to reload:

haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf $(cat /var/run/haproxy.pid)

And our config file looks like this:

global
    log 127.0.0.1 local0 notice             
    maxconn 20000                           
    user haproxy
    group haproxy
    tune.ssl.default-dh-param 2048

defaults
    log global
    mode http
    option httplog
    option dontlognull
    option http-keep-alive                  
    option forwardfor                       
    retries 3                               
    option redispatch                       
    timeout connect 5s                      
    timeout check 5s                        
    timeout client 60000                    
    timeout server 60000                    

    stats enable
    stats uri /haproxy?stats
    stats auth [REDACTED]

    option httpchk GET / HTTP/1.0           
    balance roundrobin                      
    default-server inter 10s fall 2 rise 2  

frontend http-in
        bind *:80

        # Define hosts
        acl host_1 hdr(host) -i somehost.somedomain.com
        [hundreds of host header configurations]

        ## switches
        use_backend 1 if host_1
        [hundreds of if-clauses]

frontend https-in
        bind *:443 ssl crt [REDACTED] crt [REDACTED] crt [REDACTED]

        # Define hosts
        acl host_1 hdr(host) -i somehost.somedomain.com
        [hundreds of host header configurations]

        ## switches
        use_backend 1 if host_1
        [hundreds of if-clauses]

backend 1
        server node1 [some IP] check
        server node2 [some IP] check

[lots more backends]

Thanks!

UPDATE: The only difference we have found is that the slow server uses Ubuntu 16.04, while the fast server uses 16.04.1. Not sure if that is relevant.

UPDATE 2: Some other server we have are also running 16.04 and they have fast reloads. So it's probably not that. Next step is to re-install haproxy and see if it helps.

UPDATE 3: Reinstall did not help. We're currently running with strace to try to find what it is doing. It seems like it's trying to connect to all our backends but timing out often. Not yet clear why only this server gets timeouts, and why it refuses to take over until all the polling is done.

  • 2
    Can't comment on your specific problem, but as a note: Using "hundreds of if-clauses" to select a backend based on the Host header isn't the way to go. Have a look at the mapfile directive and leverage it via something like this: use_backend %[req.hdr(host),lower,map(/etc/haproxy/domain2backend.map)]. Your mapfile should contain lines mapping a Host to a backend like the following: Host1 backend1 ...Host2 backend2...HostN backendN. – gf_ Oct 12 '17 at 10:56
  • Thanks, I suspected it wasnt the right way to do it but didnt know what to do instead. It has grown from very few to hundreds over the past years but I guess its time to fix it. – DukeOf1Cat Oct 12 '17 at 12:39
  • Please post a summary and the outcome of the thread you've started here, to help other people in the future. – gf_ Oct 13 '17 at 13:38
  • Will do. Have not yet resolved the issue. – DukeOf1Cat Oct 13 '17 at 13:46
2

It turns out that the slow start of the new HAProxy process was due to it trying to resolve the DNS's of all the backend servers. And since we had a lot of servers and a slow (or unresponsive) DNS server caused a lot of query timeouts which caused a slow start of the process.

We changed to a better DNS server and now it takes 2 seconds instead.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.