I'm new to HAProxy, but I was able to get things functional, although I am not getting the throughput that I expect.
My setup includes a 5 node storage cluster (running Riak CS, an S3 style storage) and another VM running HAProxy with
roundrobin. All VMs are using VirtualBox, each on a dedicated machine. The HAProxy VM is on Windows 7, while the others are Windows 10. This is all on the same gigabit network and I tested both 1.4 and 1.5 for HAProxy.
I have a script (python + boto) that downloads without saving to disk, 100 MB files over and over, which I believe should each boil down to a simple get request, and I run that script locally 3 times for added load.
If I point them all to the ip of HAProxy, I get approximately 4-500 Mbps download while I am able to get up to 900+ Mbps if I do NOT use HAProxy and point each to a separate ip.
When doing this test, it looks like the HAProxy is maxing out a single core and becoming bottle necked. For a single script running, which I believe is a single connection, it is eating 20-40% of a cores cpu on HAProxy, which seems suspicious?
It sounds like HAProxy can handle high throughput so I am trying to debug where I might have set things up incorrectly, either in Ubuntu or in the HAProxy config file. I see minimal improvement using
nbproc 3 in the config, the load is definitely not balanced across the 3 processes, as one is still maxed out.
Does anything about this setup, such as the VMs, jump out as potential red flag? Does my haproxy config sound culprit? Or my general Ubuntu settings? Also worth asking, is this a good or bad use case for HAProxy?
I have some further digging to do, but my current feeling is that this is VM specific, potentially in the ethernet driver (e1000)? I was able to move the HAProxy setup to a physical machine (not a VM) and on a single core, it barely registered any cpu usage with my previous test case...
global #log 127.0.0.1 local0 #log 127.0.0.1 local1 notice maxconn 256000 spread-checks 5 daemon nbproc 4 cpu-map 1 2 cpu-map 2 3 cpu-map 3 4 cpu-map 4 5 defaults option dontlog-normal option redispatch option allbackups no option httpclose retries 3 maxconn 256000 contimeout 5000 clitimeout 5000 srvtimeout 5000 option forwardfor except 127.0.0.1 frontend riak_cs bind *:8098 bind *:8080 mode http capture request header Host len 64 acl d1 dst_port 8098 acl d2 dst_port 8080 use_backend riak_cs_backend_stats if d1 use_backend riak_cs_backend if d2 backend riak_cs_backend mode http balance roundrobin option httpchk GET /riak-cs/ping timeout connect 60s timeout http-request 60s stats enable stats uri /haproxy?stats server riak1 192.168.80.105:8080 weight 1 maxconn 1024 check inter 5s server riak2 192.168.80.106:8080 weight 1 maxconn 1024 check inter 5s server riak3 192.168.80.107:8080 weight 1 maxconn 1024 check inter 5s server riak4 192.168.80.108:8080 weight 1 maxconn 1024 check inter 5s server riak5 192.168.80.109:8080 weight 1 maxconn 1024 check inter 5s backend riak_cs_backend_stats mode http balance roundrobin timeout connect 60s timeout http-request 60s stats enable stats uri /haproxy?stats server riak1 192.168.80.105:8098 weight 1 maxconn 1024 server riak2 192.168.80.106:8098 weight 1 maxconn 1024 server riak3 192.168.80.107:8098 weight 1 maxconn 1024 server riak4 192.168.80.108:8098 weight 1 maxconn 1024 server riak5 192.168.80.109:8098 weight 1 maxconn 1024