I'm trying to understand some numbers I'm seeing as well as why Haproxy seems to be hitting variable limits on requests/s.
The test set up is pretty straightforward:
- 20 backend servers running on Ubuntu VMs serving empty HTTP 200 responses
- 1 Haproxy node running on an Ubuntu VM
I ran a perf test against each of the backend servers and they're serving up ~8k req/s using the wrk tool
Putting Haproxy in front of them seems to cap out at ~22k req/s.
CPU utilization is extremely low on the Haproxy box (5-7%).
I initially suspected it might be an outbound connection limit, so I ran the wrk tool in parallel on the Haproxy box against all 20 backend servers individually. That scaled out linearly - each of the processes were able to hit ~8k req/s.
Next up I switched the backend servers for an nginx instance serving up HTTP 204s
- Hitting the nginx endpoint directly using wrk gave me ~350k req/s
- Adding Haproxy in front of it reduced this to ~35k req/s ??
I then configured Haproxy to add multiple entries for the same endpoint, i.e.
server nginx1 nginxbox:80 server nginx2 nginxbox:80
That dropped throughput down even further, to the ~25k req/s range
I've run similar experiments with Nginx hitting the backends (that caps out at ~19k/s) and running Haproxy on Centos, Haproxy versions 1.6 and the new 1.7, but the results seem to be pretty consistent. This makes me feel like there's some sort of configuration I'm missing in Haproxy, but I haven't been able to figure it out.
Note: I've specifically removed the global maxconn because for some reason adding that in seemed to increase the number of error responses
Here's the Haproxy config:
global daemon log 127.0.0.1 local0 user haproxy group haproxy defaults log global mode http option httplog option dontlognull option abortonclose retries 3 timeout connect 5000 timeout client 10000 timeout server 10000 frontend www-http log global bind *:80 default_backend myapp_api-nossl backend myapp_api-nossl http-response add-header X-App-Server %b/%s server myapp_vm-0 myapp_apivm-0:80 check maxconn 200 server myapp_vm-1 myapp_apivm-1:80 check maxconn 200 server myapp_vm-2 myapp_apivm-2:80 check maxconn 200 server myapp_vm-3 myapp_apivm-3:80 check maxconn 200 server myapp_vm-4 myapp_apivm-4:80 check maxconn 200 server myapp_vm-5 myapp_apivm-5:80 check maxconn 200 server myapp_vm-6 myapp_apivm-6:80 check maxconn 200 server myapp_vm-7 myapp_apivm-7:80 check maxconn 200 server myapp_vm-8 myapp_apivm-8:80 check maxconn 200 server myapp_vm-9 myapp_apivm-9:80 check maxconn 200 server myapp_vm-10 myapp_apivm-10:80 check maxconn 200 server myapp_vm-11 myapp_apivm-11:80 check maxconn 200 server myapp_vm-12 myapp_apivm-12:80 check maxconn 200 server myapp_vm-13 myapp_apivm-13:80 check maxconn 200 server myapp_vm-14 myapp_apivm-14:80 check maxconn 200 server myapp_vm-15 myapp_apivm-15:80 check maxconn 200 server myapp_vm-16 myapp_apivm-16:80 check maxconn 200 server myapp_vm-17 myapp_apivm-17:80 check maxconn 200 server myapp_vm-18 myapp_apivm-18:80 check maxconn 200 server myapp_vm-19 myapp_apivm-19:80 check maxconn 200