Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have an application where the client connects to a server via a simple TCP based protocol over TLS/SSL. In development, this has worked great for many months while we were building our application out. Recently as we prepare for launch, I've gone ahead and added HAProxy into the mix in order to facilitate some order of load distribution. Everything works, technically, but the issue is, the client is now seeing completely random time outs. They are not typically consistent, but happen at roughly 60 seconds long. Sometimes it can happen after 25 seconds. The server that haproxy forwards the TCP connection to notices and does a clean disconnect, the problem is you do not want a bunch of simultaneous connections getting disrupted and reconnected over and over without any reason. This has implications on our publish/subscribe infrastructure in addition to other things. The client is smart enough to reconnect right away - however that's not the behavior we desire. The server that's responsible for accepting these TCP connections over SSL does not require a keep alive. I'm going to go ahead and assume that there is some implicit value I'm not seeing in my HAProxy config causing these random timeouts, or something that requires a TCP keep alive. The fact that the timeouts are not always consistent however, makes me wonder otherwise. If it was 60 seconds on the dot each and every time I'd be convinced it's a configuration issue. In this particular case, it is not always 60 seconds. Here's what my configuration looks like right now:

global
stats socket /home/haproxy/status user haproxy group haproxy
    log 127.0.0.1   local1 info
#   log 127.0.0.1   local5 info 
    maxconn 4096
    ulimit-n 8250
        # typically: /home/haproxy
    chroot /home/haproxy
    user haproxy    
    group haproxy
    daemon
    quiet
    pidfile /home/haproxy/haproxy.pid

defaults
    log global
    mode    http
    option  httplog
    option  dontlognull
    retries 3
    option redispatch
    maxconn 2000
    contimeout  5000
    clitimeout  60000
    srvtimeout  60000

# Configuration for one application:
# Example: listen myapp 0.0.0.0:80
listen www 0.0.0.0:443
        mode tcp
        balance leastconn
    # Example server line (with optional cookie and check included)
    # server    srv3.0 10.253.43.224:8000 srv03.0 check inter 2000 rise 2 fall 3
# Status port (by default, localhost only...for debugging purposes)
    server ANID3 10.0.1.2:8888 check inter 3000 rise 2 fall 3 maxconn 500
    server ANID1 10.0.1.3:8888 check inter 3000 rise 2 fall 3 maxconn 500
    server ANID2 10.0.1.4:8888 check inter 3000 rise 2 fall 3 maxconn 500

listen health 0.0.0.0:9999
        mode http
        balance roundrobin
        stats uri /haproxy-status

I verified that HAProxy is the issue by having our client bypass it and go directly to a single app server where there is no time outs and everything is nice and dandy. As soon as I route it through one of our two haproxy servers, the random disconnects happen ranging anywhere between 25-60 seconds.

Thanks for taking a look at this. It is quite frustrating, but I'm sure it is a lack of understanding in what exactly HAProxy expects from my client.

share|improve this question
    
Version of haproxy? –  Greg Askew Jul 17 '12 at 22:44

4 Answers 4

try this, I have fixed this problem.

listen mysql-slaves
bind 0.0.0.0:3306
mode tcp
maxconn 20000
option mysql-check user haproxy
balance roundrobin
contimeout 5000
clitimeout 50000
srvtimeout 50000
....
share|improve this answer
1  
You should explain how and why that configuration solves the problem. –  kasperd Jun 9 at 10:45

Can you test this:

defaults  
    timeout client 60000  
    option http-server-close  

Instead of clitimeout

https://code.google.com/p/haproxy-docs/wiki/http_server_close

share|improve this answer

There should be no reason for early close of connection, I don't even see how that can happen. Your timeouts are set to 60s so it should be 60s.

Hmmm wait a minute, aren't you running haproxy inside a VM with a fast-running clock ? It's a problem in some VMs, where the clock sometimes runs far too fast (more than twice the correct speed) or instead too slow with large jumps once a minute. Haproxy knows how to defend against too long pauses and time jumps that it can detect, but obviously it cannot defend against clocks running too fast without being reported by the system.

If you're in a VM, you can try this :

$ while sleep 1; do date; done

And let this run for one or two minutes. Check by yourself if it's running at the correct speed. It's been a while since I last observed this nasty issue, but it does not mean it won't happen again.

BTW, you should set "option tcplog" in your TCP section and check the logs. You will then see there if from haproxy's point of view, it was a timeout, a client or server abort, and after how long a time.

share|improve this answer

As the time is variable and you have definitively confirmed the backend is not responsible, its unlikely to be a timeout setting.

Which would, oddly, lead me to a solution that perhaps being the service being restarted.

If something is restarting HAProxy on a cron (Eg. monit - which would poll every 60 seconds), then it could mean a session lasts up to 60 seconds before being terminated, or shorter.

Double check your uptime on HAProxy and if its always under a minute or so - there's your answer.

Also, it might be worth reviewing the statistics from HAProxy just to ensure that you've not hit any hard session limits causing an alternative timeout to be hit. If there are less than maxqueue requests in the queue already, for timeout queue seconds, if after that timeout no unsaturated server is found, the request will be dropped.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.