2

I'm new to HAProxy, but I was able to get things functional, although I am not getting the throughput that I expect.

My setup includes a 5 node storage cluster (running Riak CS, an S3 style storage) and another VM running HAProxy with roundrobin. All VMs are using VirtualBox, each on a dedicated machine. The HAProxy VM is on Windows 7, while the others are Windows 10. This is all on the same gigabit network and I tested both 1.4 and 1.5 for HAProxy.

I have a script (python + boto) that downloads without saving to disk, 100 MB files over and over, which I believe should each boil down to a simple get request, and I run that script locally 3 times for added load.

If I point them all to the ip of HAProxy, I get approximately 4-500 Mbps download while I am able to get up to 900+ Mbps if I do NOT use HAProxy and point each to a separate ip.

When doing this test, it looks like the HAProxy is maxing out a single core and becoming bottle necked. For a single script running, which I believe is a single connection, it is eating 20-40% of a cores cpu on HAProxy, which seems suspicious?

It sounds like HAProxy can handle high throughput so I am trying to debug where I might have set things up incorrectly, either in Ubuntu or in the HAProxy config file. I see minimal improvement using nbproc 3 in the config, the load is definitely not balanced across the 3 processes, as one is still maxed out.

Does anything about this setup, such as the VMs, jump out as potential red flag? Does my haproxy config sound culprit? Or my general Ubuntu settings? Also worth asking, is this a good or bad use case for HAProxy?

EDIT

I have some further digging to do, but my current feeling is that this is VM specific, potentially in the ethernet driver (e1000)? I was able to move the HAProxy setup to a physical machine (not a VM) and on a single core, it barely registered any cpu usage with my previous test case...

full config

global
    #log 127.0.0.1   local0
    #log 127.0.0.1   local1 notice
    maxconn 256000
    spread-checks 5
    daemon 
    nbproc 4 
    cpu-map 1 2
    cpu-map 2 3
    cpu-map 3 4
    cpu-map 4 5

defaults
    option dontlog-normal
    option  redispatch
    option  allbackups
    no option   httpclose
    retries 3
    maxconn 256000
    contimeout  5000
    clitimeout  5000
    srvtimeout  5000

    option forwardfor except 127.0.0.1

frontend riak_cs
    bind          *:8098
    bind          *:8080
    mode          http
    capture       request header Host len 64

    acl d1 dst_port 8098
    acl d2 dst_port 8080

    use_backend   riak_cs_backend_stats if d1
    use_backend   riak_cs_backend if d2



backend riak_cs_backend
    mode http 
    balance roundrobin
    option httpchk GET /riak-cs/ping
    timeout connect 60s
    timeout http-request 60s

    stats enable
    stats uri /haproxy?stats

    server riak1 192.168.80.105:8080 weight 1 maxconn 1024 check inter 5s 
    server riak2 192.168.80.106:8080 weight 1 maxconn 1024 check inter 5s
    server riak3 192.168.80.107:8080 weight 1 maxconn 1024 check inter 5s
    server riak4 192.168.80.108:8080 weight 1 maxconn 1024 check inter 5s
    server riak5 192.168.80.109:8080 weight 1 maxconn 1024 check inter 5s

backend riak_cs_backend_stats 
    mode http
    balance roundrobin
    timeout connect 60s
    timeout http-request 60s

    stats enable 
    stats uri /haproxy?stats

    server riak1 192.168.80.105:8098 weight 1 maxconn 1024 
    server riak2 192.168.80.106:8098 weight 1 maxconn 1024
    server riak3 192.168.80.107:8098 weight 1 maxconn 1024
    server riak4 192.168.80.108:8098 weight 1 maxconn 1024 
    server riak5 192.168.80.109:8098 weight 1 maxconn 1024 
  • 1
    Do you have Receive Side Scaling turned on for the NIC? – longneck Sep 13 '16 at 18:58
  • @longneck thanks for the suggestion. It was enabled in the Windows side (vm host) with just one queue, to test I set it to two but that doesn't appear to have changed anything. I need to check how this translates to the VM though – g3cko Sep 13 '16 at 19:33
  • 1
    What virtulization platform are you using? Hyper-V? What version of Windows? – longneck Sep 13 '16 at 19:35
  • Sorry forgot to add that, I'm using Virtual Box, setup with a Bridged adapter on windows 7. – g3cko Sep 13 '16 at 19:37
  • If I'm understanding this, it doesn't appear the VM side, this is setup correctly. I don't get multiple entries for eth0 from /proc/interrupts like this page shows access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/… – g3cko Sep 13 '16 at 20:02
1

I hate to answer my own question, but I think my conclusion is that my test is VM limited. I can't quite say exactly how so, but the cpu usage of HAProxy through my vm was much, much higher, and as I noted above, testing on physical hardware with the same config, even removing the nbproc part, I see a a barely noticable CPU load in HAProxy.

It wasn't my goal to run anything production through VMs, but they are much more convenient for testing (while waiting on actual hardware) and to learn how this stuff works.

| improve this answer | |
0

As you didn't showed your config nor the version in use (see my comment), this is a bit like "shooting into the dark". Anyway, you could try to pin each HAProxy process to one specific core, to try to max them out all, and to even the load between them.

Quoting the docs:

cpu-map <"all"|"odd"|"even"|process_num> <cpu-set>...

On Linux 2.6 and above, it is possible to bind a process to a specific CPU set. This means that the process will never run on other CPUs. The cpu-map directive specifies CPU sets for process sets. The first argument is the process number to bind. This process must have a number between 1 and 32 or 64, depending on the machine's word size, and any process IDs above nbproc are ignored. It is possible to specify all processes at once using all, only odd numbers using odd or even numbers using even, just like with the bind-process directive. The second and forthcoming arguments are CPU sets. Each CPU set is either a unique number between 0 and 31 or 63 or a range with two such numbers delimited by a dash ('-'). Multiple CPU numbers or ranges may be specified, and the processes will be allowed to bind to all of them. Obviously, multiple cpu-map directives may be specified. Each cpu-map directive will replace the previous ones when they overlap.

So, if you're using three processes, test the following:

cpu-map 1 0
cpu-map 2 1
cpu-map 3 2
| improve this answer | |
  • Is copy/pasting the cfg the best way to share on here? I did test using the cpu-map as above, but didn't notice any difference overall. I'll update to note that I did try on both 1.4 and 1.5 for HAProxy – g3cko Sep 13 '16 at 23:51
  • posted the cfg file above, but also added a new note from my findings – g3cko Sep 13 '16 at 23:57
  • @g3cko Sorry for the noise: This obviously only works on Linux. – gf_ Sep 14 '16 at 7:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.