I have Two HPBL685c G6 blade servers running Ubuntu 15.04

When I configure each of the 4 10GB NIC separately, I can test using iperf and I receive ~10Gbit/sec bandwidth between servers, for each NIC. This works as expected.

Now, I am trying to bond all of the 10GB NIC on each server, using bond mode "balance-rr". These results vary, but land somewhere between 2.5Gbits/sec and 5Gbits/sec

I am using the same configuration to bond 2X1GB NIC on these same servers, 2X1GB NIC bonded results in ~2Gbit/sec bandwidth when testing with iperf. These 2 NICs are not connected to a Virtual Connect Domain, and instead are each connected to a different Cisco Catalyst Blade Switch 3120

So, my question is: Why does bonding 4X10GB NIC using balance-rr, result in less performance than using a single NIC. I would have expected ~40Gbit/sec bandwidth minus TCP/Bonding overhead, which would align with my results when bonding 2X1GB and getting ~2GB when testing.

I have tried this with different bonding modes, and the others result in about ~10Gbit/sec bandwidth when bonded. Still not ideal, but better than the balance-rr results.

  • Results from cat /proc/net/bonding/bond-net show all interfaces are up with 10000Mbps d.pr/n/16hKi – Nicholas Curtis Jun 21 '15 at 0:10
  • Results from iperf -c -P 4 to test using 8 parallel clients, shows that I am able to surpass the 10Gbit/sec of a single NIC: d.pr/i/Tk3a However, I would still expect that using -P 8, each connection would get ~5Gbit/sec and SUM to ~40Gbit/sec – Nicholas Curtis Jun 21 '15 at 0:19
  • 1
    Did you try running multiple iperf streams at once? – SpacemanSpiff Jun 21 '15 at 1:50
  • Only using the -P flag, I did not start up multiple instances of iperf. However, even w/o multiple streams or parallel clients, I would expect iperf to get at least the bandwidth of a single NIC in the bond - currently it is getting 25% of one of the 4 nics when using a single iperf client. – Nicholas Curtis Jun 21 '15 at 2:10
  • Why aren't you using LACP? – ewwhite Jun 21 '15 at 4:07

It appears that the Virtual Connect Modules do not support bond mode 0 (balance-rr) in linux deployments.

From HP Support: http://h20564.www2.hp.com/hpsc/doc/public/display?docId=emr_na-c02957870

Information Unsupported bonding modes in an HP Virtual Connect environment can produce packet loss and/or performance issues.

Details HP Virtual Connect supports bonding modes 1, 5, or 6. VC does not support modes 0 (round robin) or 7 (switch assisted load balancing).

Mode 1: Active/backup. Active-backup policy: Only one slave in the bond is active. A different slave becomes active if, and only if, the active slave fails. The bond's MAC address is externally visible on only one port (network adapter) to avoid confusing the switch.

Mode 5: Adaptive transmit load balancing: channel bonding that does not require any special switch support. The outgoing traffic is distributed according to the current load (computed relative to the speed) on each slave. Incoming traffic is received by the current slave. If the receiving slave fails, another slave takes over the MAC address of the failed receiving slave.

Mode 6: Adaptive load balancing: includes balance-tlb plus receive load balancing (rlb) for IPV4 traffic, and does not require any special switch support. The receive load balancing is achieved by ARP negotiation.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.