Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I want to connect a CentOS 6.4 Linux Box with two NICs to a Cisco 2960S using LACP 802.3ad port aggregation. This mainly for redundancy reasons (and hopefully more bandwith). We don't use VLAN tagging.

With the config listed below the link aggregation only works partially. Approx half of the network hosts can ping and ssh the Linux box, whereas the other half cannot. Same is true for the Linux box itself, approx only half of the hosts can be pinged.

Setting up adapter bonding (or in Cisco speech EtherChannel) shouldn't be that hard. But does anyone know what's wrong here?

On the Linux side, the configuration looks like this:

cat /etc/modprobe.d/bond.conf 
alias bond0 bonding  

cat /etc/sysconfig/network-scripts/ifcfg-bond0 
DEVICE=bond0
ONBOOT=yes
USERCTL=no
BOOTPROTO=none
NM_CONTROLLED="no"
IPADDR=10.76.161.135
PREFIX=21
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System bond0"
BONDING_OPTS="mode=4 miimon=100 lacp_rate=1"

cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE="eth0"
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no

cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE="eth1"
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no

And this commands I applied to the Cisco 2960S:

sw01>enable     
sw01#config term
sw01(config)#int range Gi0/13 - 14
sw01(config-if-range)#description lacp ch2     
sw01(config-if-range)#channel-protocol lacp
sw01(config-if-range)#channel-group 2 mode active
Creating a port-channel interface Port-channel 2
sw01(config-if-range)#no shutdown
sw01(config-if-range)#exit
sw01(config)#interface Port-channel2
sw01(config-if)#description lacp ch2 for ssensvr03
sw01(config-if)#switchport mode access
sw01(config-if)#no shutdown
sw01(config-if)#exit

sw01>show interface description 
Gi0/13                         up             up       lacp ch2
Gi0/14                         up             up       lacp ch2
Po2                            up             up       lacp ch2 for svr03
sw01>show etherchannel summary
Number of channel-groups in use: 1
Number of aggregators:           1

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
2      Po2(SU)         LACP      Gi0/13(P)   Gi0/14(P)   

sw01>show etherchannel 
Group: 2 
----------
Group state = L2 
Ports: 2   Maxports = 16
Port-channels: 1 Max Port-channels = 16
Protocol:   LACP
Minimum Links: 0

For the sake of completeness some output on the Linux side:

ifconfig
bond0     Link encap:Ethernet  HWaddr E8:39:35:B8:08:4C  
          inet addr:10.76.161.135  Bcast:10.76.167.255  Mask:255.255.248.0
          inet6 addr: fe80::ea39:35ff:feb8:84c/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:8925 errors:0 dropped:0 overruns:0 frame:0
          TX packets:584 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:997292 (973.9 KiB)  TX bytes:78966 (77.1 KiB)

eth0      Link encap:Ethernet  HWaddr E8:39:35:B8:08:4C  
          inet addr:10.76.161.135  Bcast:10.76.167.255  Mask:255.255.248.0
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:5458 errors:0 dropped:0 overruns:0 frame:0
          TX packets:447 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:564784 (551.5 KiB)  TX bytes:62187 (60.7 KiB)
          Memory:fbd00000-fbe00000 

eth1      Link encap:Ethernet  HWaddr E8:39:35:B8:08:4D  
          inet addr:10.76.161.135  Bcast:10.76.167.255  Mask:255.255.248.0
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:3467 errors:0 dropped:0 overruns:0 frame:0
          TX packets:137 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:432508 (422.3 KiB)  TX bytes:16779 (16.3 KiB)
          Memory:fbb00000-fbc00000

route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.76.160.0     *               255.255.248.0   U     0      0        0 bond0
10.76.160.0     *               255.255.248.0   U     1      0        0 eth1
10.76.160.0     *               255.255.248.0   U     1      0        0 eth0
link-local      *               255.255.0.0     U     1004   0        0 bond0

cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: fast
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 2
        Actor Key: 17
        Partner Key: 2
        Partner Mac Address: a4:18:75:bb:d3:00

Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: e8:39:35:b8:08:4c
Aggregator ID: 1
Slave queue ID: 0

Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: e8:39:35:b8:08:4d
Aggregator ID: 1
Slave queue ID: 0
share|improve this question
add comment

2 Answers

I wonder if this is because you are setting Fast LACPDUs (lacp_rate=1) on the Linux end of the bond, but the switch is still running in the default Slow LACPDUs mode (the default), so the bond isn't negotiating properly.

If this is right, you'll be able to either show etherchannel 2 detail or show lacp internal on the switch, the flags on the Channel Group will probably say SA (Slow Active). If you do a show lacp neigh you'll probably see F on the Linux end (Fast).

To resolve this, just remove lacp_rate=1 from your BONDING_OPTS and restart.

Everything else is configured correctly, though you don't need alias bond0 bonding, the network scripts will load and configure the bonding driver when starting the interface.

share|improve this answer
    
That was indeed a configuration error. But the problem still persists after this correction. –  StackUnderflow Jul 1 '13 at 14:48
add comment
up vote 0 down vote accepted

RHEL and CentOS have the NetworkManager enabled by default, which causes troubles. Permanently disable it as root in order to make your adapter bonding working properly:

service NetworkManager stop
chkconfig NetworkManager off
chkconfig network on
service network restart

Additionally to this remove the lacp_rate=1 from the BONDING_OPTS:

BONDING_OPTS="mode=4 miimon=100"
share|improve this answer
    
You can also just add NM_CONTROLLED=no to any ifcfg-files for interfaces you don't want managed by NM. I do this on my laptop, so network scripts manage the bridge and wired connection, but I can still use wifi from the graphical interface. –  suprjami Jan 9 at 10:10
    
In fact you should do this, because starting NM in the graphical interface with nm-applet will let NM manage any interfaces which don't have NM_CONTROLLED=no anyway. –  suprjami Jan 9 at 10:11
1  
suprjami, you're right. NM_CONTROLLED=no is the better solution in most cases. –  StackUnderflow Apr 25 at 7:57
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.