I'm seeing some confusing behaviour regarding bonded interfaces under Linux and I'd like to throw the situation out there in hopes that someone can clear it up for me.
I have two servers: Server 1 (S1) has 4x 1Gbit ethernet connections; Server 2 (S2) has 2x 1Gbit ethernet connections. Both servers are running Ubuntu 12.04, albeit with kernel 3.11.0-15 (from the lts-saucy linux-generic package).
Both servers have all their respective network interfaces bundled into a single bond0 interface with the following configuration (in
bond-mode 802.3ad bond-miimon 100 bond-lacp-rate fast bond-slaves eth0 eth1 [eth2 eth3]
Between the servers are a couple of HP switches which are (I think) correctly configured for LACP on the ports in question.
Now, the link is working - network traffic flows happily to and from both machines. And all respective interfaces are being used, so it's not like the aggregation is completely failing. However, I need as much bandwidth as possible between these two servers, and I'm not getting the ~2Gbit/s that I would expect.
In my testing, I can observe that each server seems to allocate each TCP connection (e.g. iperf, scp, nfs, whatever) to a single slave interface. Essentially everything seems capped at a max of 1 gigabit.
bond-xmit-hash-policy layer3+4, I can use
iperf -c S1 -P2 to send on two slave interfaces, but on the server side, reception is still only occurring on one slave interface and the total throughput is therefore capped at 1Gbit/s, i.e. the client shows ~40-50MB/s on two slave interfaces, the server shows ~100MB/s on one slave interface. Without setting
bond-xmit-hash-policy the sending is also limited to one slave interface.
I was under the impression that LACP should allow this kind of connection bundling, allowing, for example, a single scp transfer to make use of all available interfaces between the two hosts.
Is my understanding of LACP wrong? Or have I missed some configuration options somewhere? Any suggestions or clues for investigation would be much appreciated!