I'm running into problems with getting a LACP trunk to operate properly on Ubuntu 12.04.2 LTS.
My setup is a single host connected with two 10 Gbe interfaces to two seperate Nexus 5548 switches, with vPC configured to enabled multi-chassis LACP. Nexus config is as per Cisco guidelines, and Ubuntu config as per https://help.ubuntu.com/community/UbuntuBonding
Server is connected to port Ethernet1/7 on each Nexus switch, whose ports are configured identical and placed in Port-channel 15. Port-channel 15 is configured as VPC 15, and VPC output looks good. These are simple access ports, i.e. no 801.1q trunking involved.
+----------+ +----------+ +----------+ +----------+ | client 1 |------| nexus 1 |------| nexus 2 |------| client 2 | +----------+ +----------+ +----------+ +----------+ | | | +--------+ | +----| server |----+ eth4 +--------+ eth5
When either link is down, both clients 1 and 2 are able to reach the server. However, when I bring the secondary link up, the client connected to the switch with the newly-enabled link, is unable to reach the server. See the following table for state transitions and results:
port states (down by means of "shutdown") nexus 1 eth1/7 up up down up nexus 2 eth1/7 down up up up connectivity client 1 - server OK OK OK FAIL client 2 - server OK FAIL OK OK
Now, I belive I've isolated the issue to the Linux side. When in up-up state, each nexus uses the local link to the server to deliver the packets, as verified by looking at the mac address table. What I am able to see on the server is that the packets from each client are being received on the ethX interface (packets from client 1 on eth4, packets from client 2 on eth4) by using tcpdump -i ethX, but when I run tcpdump -i bond0 I can only traffic from either of the host (in accordance with what I stated above).
I observe the same behaviour for ARP and ICMP (IP) traffic; ARP fails from a client when both links are up, works (along with ping) when one is down, ping fails when I enable the link again (packets are still received on eth interface, but not on bond0).
To clarify, I'm setting up multiple servers in this configuration, and all show the same symptoms, so it doesn't appear to be hardware related.
So - figuring out how to fix that is what I'm dealing with; my Googling has not brought me any luck so far.
Any pointers are highly appreciated.
auto eth4 iface eth4 inet manual bond-master bond0 auto eth5 iface eth5 inet manual bond-master bond0 auto bond0 iface bond0 inet static address 10.0.11.5 netmask 255.255.0.0 gateway 10.0.0.3 mtu 9216 dns-nameservers 22.214.171.124 126.96.36.199 bond-mode 4 bond-miimon 100 bond-lacp-rate 1 #bond-slaves eth4 bond-slaves eth4 eth5
A little further information: Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer2 (0) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 802.3ad info LACP rate: fast Min links: 0 Aggregator selection policy (ad_select): stable Active Aggregator Info: Aggregator ID: 1 Number of ports: 1 Actor Key: 33 Partner Key: 1 Partner Mac Address: 00:00:00:00:00:00 Slave Interface: eth4 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 8 Permanent HW addr: 90:e2:ba:3f:d1:8c Aggregator ID: 1 Slave queue ID: 0 Slave Interface: eth5 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 13 Permanent HW addr: 90:e2:ba:3f:d1:8d Aggregator ID: 2 Slave queue ID: 0
EDIT: Added config from Nexus
vpc domain 100 role priority 4000 system-priority 4000 peer-keepalive destination 10.141.10.17 source 10.141.10.12 peer-gateway auto-recovery interface port-channel15 description server5 switchport access vlan 11 spanning-tree port type edge speed 10000 vpc 15 interface Ethernet1/7 description server5 internal eth4 no cdp enable switchport access vlan 11 channel-group 15
EDIT: Added results from non-VPC port-channel on nexus1 for same server, before and after IP change (changed IP to influence load-balancing algorithm). This is still using the same settings on the server.
port states (down by means of "shutdown") nexus 1 eth1/7 up up down up nexus 1 eth1/14 down up up up <= port moved from nexus 2 eth1/7 connectivity (sever at 10.0.11.5, hashing uses Eth1/14) client 1 - server OK OK OK FAIL client 2 - server OK OK OK FAIL
The results after changing the IP is as predicted; not-used interface being brought up causes failures.
connectivity (sever at 10.0.11.15, hashing uses Eth1/7) client 1 - server OK FAIL OK OK client 2 - server OK FAIL OK OK