I'm having issues with my haproxy servers rejecting new connections (or timing them out) after a certain threshold. The proxy servers are AWS c5.large EC2's with 2 cpus and 4GB of ram. The same configuration is used for both connection types on our site, we have one for websocket connections which typically have between 2K-4K concurrent connections and a request rate of about 10/s. The other is for normal web traffic with nginx as the backend with about 400-500 concurrent connections and a request rate of about 100-150/s. Typical cpu usage for both is about 3-5% on the haproxy process, with 2-3% of the memory used for the websocket proxy (40-60MB) and 1-3% of the memory used for the web proxy (30-40MB).
Per the attached config, the cpus are mapped across both cpus, with one process and two threads running. Both types of traffic are typically 95% (or higher) SSL traffic. I've watched the proxy info using watch -n 1 'echo "show info" | socat unix:/run/haproxy/admin.sock -' to see if I'm hitting any of my limits, which does not seem to be the case.
During high traffic time, and when we start to see issues, is when our websocket concurrent connections gets up to about 5K and web requests rate gets up to 400 requests/s. I mention both servers here because I know the config can handle the high concurrent connections and request rate, but I'm missing some other resource limit being reached. Under normal conditions everything works just fine; however, the issues we see are ERR_CONNECTION_TIMED_OUT (from chrome) type errors. Never do I see any 502 errors. Nor do I see any other process use more cpu or memory on the server. I'm also attaching some other possibly relevant configs, such as setting my limits and sysctl settings.
Any ideas what I might be missing? Am I reading top and ps aux | grep haproxy wrong and seeing the wrong cpu/mem usage? Am I missing some tcp connection limit? The backend servers (nginx/websocket) are being worked, but never seem to be taxed. We've load tested these with much more connections and traffic and are limited by the proxy long before we limit the backend servers.
Thanks a lot.
global ulimit-n 300057 quiet maxconn 150000 maxconnrate 1000 nbproc 1 nbthread 2 cpu-map auto:1/1-2 0-1 daemon stats socket /run/haproxy/admin.sock mode 600 level admin stats timeout 2m log 127.0.0.1:514 local0 ca-base /etc/ssl/certs crt-base /etc/ssl/private ssl-default-bind-options no-sslv3 no-tlsv10 ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL:!RC4 defaults maxconn 150000 mode http log global option forwardfor timeout client 30s timeout server 120s timeout connect 10s timeout queue 60s timeout http-request 20s frontend default_proxy option httplog bind :80 bind :443 ssl crt /etc/haproxy/ssl.pem ... acl stuff which may route to a different backend ... acl for websocket traffic use_backend websocket if websocket_acl default_backend default_web backend default_web log global option httpclose option http-server-close option checkcache balance roundrobin option httpchk HEAD /index.php HTTP/1.1\r\nHost:website.com server web1 192.168.1.2:80 check inter 6000 weight 1 server web2 192.168.1.3:80 check inter 6000 weight 1 backend websocket # no option checkcache option httpclose option http-server-close balance roundrobin server websocket-1 192.168.1.4:80 check inter 6000 weight 1 server websocket-2 192.168.1.5:80 check inter 6000 weight 1
Output from haproxy -vv:
HA-Proxy version 1.8.23-1ppa1~xenial 2019/11/26 Copyright 2000-2019 Willy Tarreau <email@example.com> Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -O2 -g -O2 -fPIE -fstack-protector-strong -Wformat - Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_NS=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016 Running on OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 Built with Lua version : Lua 5.3.1 Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Encrypted password support via crypt(3): yes Built with multi-threading support. Built with PCRE2 version : 10.21 2016-01-12 PCRE2 library supports JIT : yes Built with zlib version : 1.2.8 Running on zlib version : 1.2.8 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with network namespace support. Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Available filters : [SPOE] spoe [COMP] compression [TRACE] trace
* soft nofile 120000 * soft nproc 120000
net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_syncookies=1 net.ipv4.conf.all.rp_filter = 1 net.ipv4.ip_local_port_range = 1024 65023 net.ipv4.tcp_max_syn_backlog = 50000 net.ipv4.tcp_max_tw_buckets = 400000 net.ipv4.tcp_max_orphans = 60000 net.ipv4.tcp_synack_retries = 3 net.core.somaxconn = 50000 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_wmem = 4096 65536 16777216 net.ipv4.tcp_no_metrics_save = 1 net.ipv4.tcp_moderate_rcvbuf = 1 net.core.netdev_max_backlog = 50000 fs.epoll.max_user_instances = 10000
Typical with load with 330 concurrent connections and 80 req/s ps aux | grep haproxy output:
root 8122 4.5 1.2 159052 46200 ? Ssl Jan28 40:56 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 29790 root 12893 0.0 0.3 49720 12832 ? Ss Jan21 0:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 29790
and the OS is Ubuntu 16.04.