2

I have HAProxy 1.5 running on Ubuntu 14.04 (modified). It accepts connections on http and https ports. Two backend applications process requests using persistent connection.

When I create around 2200 client connections haproxy stops accepting additional connections. But I want this system to accept at least 10K simultaneous connections.

Here is connection statistics:

# ss -s
TCP:   4119 (estab 4098, closed 6, orphaned 0, synrecv 0, timewait 6/0), ports 0

Transport Total     IP        IPv6
TCP       4113      4106      7
INET      4128      4117      11

I have already tuned maximum number of opened files for the process:

# cat /proc/1012/limits
Limit                     Soft Limit           Hard Limit           Units
Max open files            240017               240017               files

My haproxy.config file:

global
    log /dev/log syslog debug
    daemon
    user haproxy
    group haproxy
    maxconn 120000
    spread-checks 4

defaults
    log global
    timeout connect 30000ms
    timeout client 300000ms
    timeout server 300000ms

frontend http-in
    mode http
    bind :80
    option httplog
    option forwardfor
    reqadd X-Forwarded-Proto:\ http
    default_backend http-routers

frontend https-in
    mode http
    bind :443 ssl crt /opt/haproxy/cert.pem no-sslv3
    option httplog
    option forwardfor
    option http-server-close
    reqadd X-Forwarded-Proto:\ https
    default_backend http-routers

frontend ssl-in
    mode tcp
    bind :4443 ssl crt /opt/haproxy/cert.pem no-sslv3
    default_backend tcp-routers

backend http-routers
    mode http
    balance roundrobin
        server node0 192.168.10.2:80 check inter 1000
        server node1 192.168.10.2:80 check inter 1000

backend tcp-routers
    mode tcp
    balance roundrobin
        server node0 192.168.10.2:80 check inter 1000
        server node1 192.168.10.2:80 check inter 1000
5
  • 1
    Please post your haproxy.cfg. Feb 18, 2015 at 14:51
  • I have added my configuration to the post
    – lexsys
    Feb 18, 2015 at 15:02
  • The configuration doesn't look like it would be an issue. Why do you think haproxy is the bottleneck? Try checking dmesg, see if that's giving you any relevant information. Feb 18, 2015 at 15:15
  • Did you try to increase somaxconn just in case you spawn connections too fast? Feb 18, 2015 at 15:46
  • I can't established a single additional connection when the 2.2K limit is reached, so I think connection rate is not limiting factor. I tried to increased net.core.somaxconn = 1024, but it didn't work.
    – lexsys
    Feb 18, 2015 at 17:33

3 Answers 3

2

As far as I know, listen block maxconn is different then global maxconn. With global maxconn you limit the max number of connections you let the haproxy process handle.

Listen / frontend section has its own maxconn, which limits the nubmer of connections per listener. So, try to set up maxconn in your frontend sections too, or at least set it up in default section.

So either:

 defaults
     maxconn 10000

or set it up per frontend.

1
  • 1
    Thank you, Jakov! This solution helped me - now I have 16K connections established.
    – lexsys
    Feb 18, 2015 at 17:44
0

What is the hardware configuration of your HAProxy machine? Cores, RAM, network interfaces, etc?

We have run into many HAProxy performance issues. Here are some ideas:

  1. Split traffic into two HAProxy servers, with each acting as a backup for the other.

  2. Raise nbproc to reflect the number of cores on your particular hardware.

  3. If you are using layer 7 mode, see if you can drop down to layer 4 mode for more throughput.

0

Haproxy has changed since 1.5 version, but as the management guide points on section "5. File-descriptor limitations", these are the limits that you can encounter:

1- Max number of file descriptors. Automatically set at startup by Haproxy.

Rule maxconn value x 2 ~= max file descriptors

On conf file:

maxconn 500000

Form web stats report:

maxconn = 500.000
maxsock = 1.000.070

And from system:

# pidof haproxy
# cat /proc/630/limits | grep 'open files'
Limit                     Soft Limit             Hard Limit             Units 
Max open files            1.000.070              1.000.070              files

Identificable error -> strace accept() or socket() return "-1 EMFILE"

2- System-wide file descriptors. Set at boot based on the amount of memory

# cat /proc/sys/fs/file-max
3.074.560

Identificable error -> strace accept() or socket() return "-1 ENFILE"

3- Per-process hard limit on the number of file descriptors.

# cat /proc/sys/fs/nr_open 
1.048.576

Identificable error -> strace accept() or socket() return "-1 ENFILE"

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .