I have Two HPBL685c G6 blade servers running Ubuntu 15.04
- Each server has 4X10GB NIC
- 2x10GB NIC are connected to a single VirtualConnect 10/10G Ethernet module
- The other 2x10GB NIC are connected to a second VirtualConnect 10/10G Ethernet module
- Virtual connect modules are configured for Horizontal Stacking, and are located in interconnect bays 1 & 2
- The NICs being referenced are all embedded Flex-10 adapters
When I configure each of the 4 10GB NIC separately, I can test using iperf and I receive ~10Gbit/sec bandwidth between servers, for each NIC. This works as expected.
- Server 1: http://d.pr/n/15dA5
- Server 2: http://d.pr/n/10pqJ
- iperf results: http://d.pr/i/pscU and http://d.pr/i/zh2E
Now, I am trying to bond all of the 10GB NIC on each server, using bond mode "balance-rr". These results vary, but land somewhere between 2.5Gbits/sec and 5Gbits/sec
- Server 1: http://d.pr/n/1aTei
- Server 2: http://d.pr/n/12Mqy
- iperf results: http://d.pr/i/1cvh0 and http://d.pr/i/1eOgU
I am using the same configuration to bond 2X1GB NIC on these same servers, 2X1GB NIC bonded results in ~2Gbit/sec bandwidth when testing with iperf. These 2 NICs are not connected to a Virtual Connect Domain, and instead are each connected to a different Cisco Catalyst Blade Switch 3120
- Server 1: http://d.pr/n/1kk4f
- Server 2: http://d.pr/n/fbVJ
- iperf results: http://d.pr/i/10N4q and http://d.pr/i/1a0z3
So, my question is: Why does bonding 4X10GB NIC using balance-rr, result in less performance than using a single NIC. I would have expected ~40Gbit/sec bandwidth minus TCP/Bonding overhead, which would align with my results when bonding 2X1GB and getting ~2GB when testing.
I have tried this with different bonding modes, and the others result in about ~10Gbit/sec bandwidth when bonded. Still not ideal, but better than the balance-rr results.