I'm in the process of setting up a series of servers as proxies to our application server, primarily to obscure the application server's IP as part of a suite of anti-DDOS measures.
My goal is to have HAProxy accept short-lived HTTP/1.0 connections and forward them to the backend over persistent connections, so as to significantly cut down on the overhead lost to the connection process, slow start, etc.
I have this configuration partly working, but am still seeing a much higher connection rate to the backend than expected - on investigation, it appears that each backend connection is serving between 1 and 5 requests, then being reset by the HAProxy side.
These reset packets always happen after a response is fully received from the server, before any further request is sent, and they happen after a seemingly random delay - sometimes as short as 0.1s, sometimes as long as 20s. Changing HAProxy's timeout settings seems to have no effect.
Does anyone know what would cause HAProxy to exhibit this behaviour, or how I might better go about debugging it?
Edit: Forgot to mention, I was unable to get the backends to generate any meaningful logs, even using tcplog
global log /dev/log local0 log /dev/log local1 notice stats socket /run/haproxy/admin.sock mode 660 level admin stats timeout 30s maxconn 16384 chroot /var/lib/haproxy user haproxy group haproxy daemon defaults log global mode http option httplog option dontlognull option http-keep-alive timeout connect 5s timeout client 50s timeout server 50s timeout http-keep-alive 50s timeout http-request 30s maxconn 4000 errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http # Temporary setting, egress is fragile w/ high rate of connections default-server maxconn 200 frontend fe-web bind 0.0.0.0:80 acl cloudflare src -f /etc/haproxy/cloudflare_ips block if !cloudflare use_backend be-web frontend fe-legacy bind 0.0.0.0:29080 option forwardfor header CF-Connecting-IP use_backend be-legacy backend be-web server srv-web 184.108.40.206:34020 check option httpchk HEAD / HTTP/1.1\r\nHost:\ legacy-domain backend be-legacy server srv-legacy 220.127.116.11:34020 check option httpchk HEAD / HTTP/1.1\r\nHost:\ legacy-domain