Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have an application where the client connects to a server via a simple TCP based protocol over TLS/SSL. In development, this has worked great for many months while we were building our application out. Recently as we prepare for launch, I've gone ahead and added HAProxy into the mix in order to facilitate some order of load distribution. Everything works, technically, but the issue is, the client is now seeing completely random time outs. They are not typically consistent, but happen at roughly 60 seconds long. Sometimes it can happen after 25 seconds. The server that haproxy forwards the TCP connection to notices and does a clean disconnect, the problem is you do not want a bunch of simultaneous connections getting disrupted and reconnected over and over without any reason. This has implications on our publish/subscribe infrastructure in addition to other things. The client is smart enough to reconnect right away - however that's not the behavior we desire. The server that's responsible for accepting these TCP connections over SSL does not require a keep alive. I'm going to go ahead and assume that there is some implicit value I'm not seeing in my HAProxy config causing these random timeouts, or something that requires a TCP keep alive. The fact that the timeouts are not always consistent however, makes me wonder otherwise. If it was 60 seconds on the dot each and every time I'd be convinced it's a configuration issue. In this particular case, it is not always 60 seconds. Here's what my configuration looks like right now:

stats socket /home/haproxy/status user haproxy group haproxy
    log   local1 info
#   log   local5 info 
    maxconn 4096
    ulimit-n 8250
        # typically: /home/haproxy
    chroot /home/haproxy
    user haproxy    
    group haproxy
    pidfile /home/haproxy/

    log global
    mode    http
    option  httplog
    option  dontlognull
    retries 3
    option redispatch
    maxconn 2000
    contimeout  5000
    clitimeout  60000
    srvtimeout  60000

# Configuration for one application:
# Example: listen myapp
listen www
        mode tcp
        balance leastconn
    # Example server line (with optional cookie and check included)
    # server    srv3.0 srv03.0 check inter 2000 rise 2 fall 3
# Status port (by default, localhost only...for debugging purposes)
    server ANID3 check inter 3000 rise 2 fall 3 maxconn 500
    server ANID1 check inter 3000 rise 2 fall 3 maxconn 500
    server ANID2 check inter 3000 rise 2 fall 3 maxconn 500

listen health
        mode http
        balance roundrobin
        stats uri /haproxy-status

I verified that HAProxy is the issue by having our client bypass it and go directly to a single app server where there is no time outs and everything is nice and dandy. As soon as I route it through one of our two haproxy servers, the random disconnects happen ranging anywhere between 25-60 seconds.

Thanks for taking a look at this. It is quite frustrating, but I'm sure it is a lack of understanding in what exactly HAProxy expects from my client.

share|improve this question
Version of haproxy? – Greg Askew Jul 17 '12 at 22:44

try this, I have fixed this problem.

listen mysql-slaves
mode tcp
maxconn 20000
option mysql-check user haproxy
balance roundrobin
contimeout 5000
clitimeout 50000
srvtimeout 50000
share|improve this answer
You should explain how and why that configuration solves the problem. – kasperd Jun 9 '14 at 10:45

As the time is variable and you have definitively confirmed the backend is not responsible, its unlikely to be a timeout setting.

Which would, oddly, lead me to a solution that perhaps being the service being restarted.

If something is restarting HAProxy on a cron (Eg. monit - which would poll every 60 seconds), then it could mean a session lasts up to 60 seconds before being terminated, or shorter.

Double check your uptime on HAProxy and if its always under a minute or so - there's your answer.

Also, it might be worth reviewing the statistics from HAProxy just to ensure that you've not hit any hard session limits causing an alternative timeout to be hit. If there are less than maxqueue requests in the queue already, for timeout queue seconds, if after that timeout no unsaturated server is found, the request will be dropped.

share|improve this answer

Can you test this:

    timeout client 60000  
    option http-server-close  

Instead of clitimeout

share|improve this answer

There should be no reason for early close of connection, I don't even see how that can happen. Your timeouts are set to 60s so it should be 60s.

Hmmm wait a minute, aren't you running inside a VM with a fast-running clock ? It's a problem in some VMs, where the clock sometimes runs far too fast (more than twice the correct speed) or instead too slow with large jumps once a minute. Haproxy knows how to defend against too long pauses and time jumps that it can detect, but obviously it cannot defend against clocks running too fast without being reported by the system.

If you're in a VM, you can try this :

$  while sleep 1; do date; done

And let this run for one or two minutes. Check by yourself if it's running at the correct speed. It's been a while since I last observed this nasty issue, but it does not mean it won't happen again.

BTW, you should set "option tcplog" in your TCP section and check the logs. You will then see there if from haproxy's point of view, it was a timeout, a client or server abort, and after how long a time.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.