9

I have 2 linux boxes running centos 6.5 each with 2 interfaces bonded together, linked to a Cisco 2960-S switch with lacp configured ports.

The configuration on the switch

port-channel load-balance src-dst-mac
!
interface Port-channel1
 switchport access vlan 100
 switchport mode access
!
interface Port-channel2
 switchport access vlan 100
 switchport mode access
!
interface FastEthernet0
 no ip address
!
interface GigabitEthernet0/1
 switchport access vlan 100
 switchport mode access
 speed 1000
 duplex full
 spanning-tree portfast
 channel-protocol lacp
 channel-group 1 mode active
!
interface GigabitEthernet0/2
 switchport access vlan 100
 switchport mode access
 speed 1000
 duplex full
 spanning-tree portfast
 channel-protocol lacp
 channel-group 1 mode active
!
interface GigabitEthernet0/3
 switchport access vlan 100
 switchport mode access
 speed 1000
 duplex full
 spanning-tree portfast
 channel-protocol lacp
 channel-group 2 mode active
!
interface GigabitEthernet0/4
 switchport access vlan 100
 switchport mode access
 speed 1000
 duplex full
 spanning-tree portfast
 channel-protocol lacp
 channel-group 2 mode active
!

and on the both linux sides I've loaded the kernel bonding module with the configuration

alias bond0 bonding
options bond0 miimon=100 mode=4 lacp_rate=1

Now the problem is that I transfer many files from one server to another monitoring the traffic graphs showing that the speed doesn't exceed the 1Gb/s speed for the bonding interface bond0.

is there any problem with the configuration ? shouldn't the speed be doubled to 2Gb/s ?

4 Answers 4

12

LACP will not split packets across multiple interfaces for a single stream/thread. For example a single TCP stream will always send/receive packets on the same NIC.

See the following post for reference:

Link aggregation (LACP/802.3ad) max throughput

Hope this helps.

9
  • Well, I didn't transfer only one file, many scp instances were copying files from one server to another.Also considering the hashing algorithm I ran a copy instance from a third server to one with bonding interface. This didn't work either and the traffic is still going through only one of the bonded interfaces. Jan 24, 2014 at 20:22
  • Are you actually using vLANs as well? I noticed that you had both port groups with access to the same vLAN. Are the other servers configured for this vLAN as well? Also what kind of switch are the devices connected to? Is it a managed switch that can handle LACP port groups and are the ports for the server with LAGs configured in a Link Aggregate on the switch? Jan 24, 2014 at 20:27
  • Server A have eth0->Gi0/1 on the switch and eth1->Gi0/2, Server B : eth0->Gi0/3 and eth1->Gi0/4. You can consider that they are in the same lan as the four ports are configured on the same vlan. The switch as I mentioned is Cisco Catalyst 2960-S configured with LACP for each couple of ports.You can see that in the configuration above. Jan 24, 2014 at 20:36
  • Sorry, long day and I totally missed the switch model. What is the speed if you transfer a single file? Is the single file transfer at 1Gb/s as well? Jan 24, 2014 at 20:47
  • also have you tried changing the balancing algorithm? Maybe try 'port-channel load-balance src-dst-port' and see if this is any different. Depending on the type of traffic you may see a different result. Jan 24, 2014 at 20:52
7

The way Link Aggregation works is by using a hashing algorithm to decide which packets should go out which port.

Packets from the same source MAC address, to the same destination MAC address, will always go out the same port.

Some Link Aggregation implementations support using layer 3 (IP addresses) and even layer 4 (TCP/UDP Port number) as part of the hash, but this is not that common.

This is why you only get 1 gbit/sec when transferring files from 1 server to another.

If the OS and Switch will both support layer 3, you can get more speed by using multiple IP addresses. However because of the way the hashing algorithm works, there is a 50/50 chance that both streams will end up going out the same link.

1
  • You can also use layer3+4 xmit_hash_policy, so that IP+port is used.
    – Marki555
    Jun 20, 2018 at 11:46
4

My understanding of network bonding is that you cannot exceed the link speed of the member interfaces in one connection. A connection will stick to one interface in the bond after it is established.

However, connections are now split between the two interfaces. If you were to have two connections running from server A to server B, then the connections shouldn't start bottle-necking each other as far as bandwidth goes because they will be traveling across different interfaces. Your total bandwidth using multiple connections should be 2Gb/s, but each connection will be limited to a maximum of 1Gb/s.

1

Its also worth considering the hash algorithm being used by Linux. Some versions of the bond driver use very simplistic hash algorithms e.g l2 hash on linux 3.6.5 is just the xor of the last byte of the source and destination MAC which leads to unbalanced traffic in a lot of circumstances. Changing the hash algorithm to l2+l3 will help a lot.

1
  • OK so the Cisco switch would receive on two interfaces, fine. How the switch would know to transmit utilizing two interfaces?
    – kubanczyk
    Sep 20, 2016 at 15:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .