Disable internal Intel X710 LLDP agent

Question

We do special hardware configurations that require the heavy use of LLDP. We have a few new racks of servers that all use the Intel X710 10Gb network card. LLDP suddenly stopped working. Our implementation of LLDP is simple. Enable LLDP on the TOR (top of rack) switch using default TLVs. Enable LLDP on the Linux image using lldpad (CentOS 6.5) and use lldptool to extract neighbor information, which has worked for thousands of machines in the past. Only, for these machines with these NICs, the whole thing just stopped working.

Use of packet dumps from the switches and the server showed that frames were properly sent to the switch from the servers and conversely, the switches were properly receiving frames from the servers and sending TLV frames back to the servers. The servers were not receiving the switch frame TLVs, though, leaving us scratching our heads. We placed other machines using different NICs on the TOR and they get LLDP data as expected.

I asked the Googles...

According to this link it seems that these X710s are probably running an internal LLDP agent, which is intercepting LLDP frames from the switch. The firmware on the affected machines we're seeing this occur is:

# ethtool -i eth2
driver: i40e
version: 1.3.47
firmware-version: 4.53 0x80001e5d 17.0.10
bus-info: 0000:01:00.2
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

The method to disable the internal LLDP agent on the NIC does not work. Nevertheless, I'm still digging around, but I figure I have a few options:

1. find the correct way to disable the internal LLDP agent on the NIC and use the existing method of extracting LLDP data on these machines -- preferred.
2. Use the NIC LLDP agent and find a way to extract the neighbor TLVs from the NIC.

Has anyone else experienced the same or similar issues with these cards and if so, how did you get around the problem?

I figure that if I wanted to use the internal agent data that it would be exposed via ethtool or snmp, but I have been unsuccessful as yet at finding a way to surface the information.

TIA

EDIT For the record, when I attempt the steps outlined in the Intel forums, I get the following output:

root@host (~)# find /sys/kernel/debug/
/sys/kernel/debug/
root@host (~)# mkdir /sys/kernel/debug/i40e
mkdir: cannot create directory `/sys/kernel/debug/i40e': No such file or directory

Answer 1

OK. So the Googles came through for me. Here's how to fix the issue.

Turns out that in order to use the debug filesystem, it needs to be mounted first. We're using a memfs OS to run commands on the machines we're tuning and by default we don't mount debugfs. So this script gave me the answer I needed.

...and the following steps for my use case worked:

root@host (~)# mount -t debugfs none /sys/kernel/debug
root@host (~)# echo lldp stop > /sys/kernel/debug/i40e/0000:01:00.2/command

yielding:

root@host (~)# lldptool -i eth2 stat
Total Frames Transmitted        = 1834
Total Discarded Frames Received = 0
Total Error Frames Received     = 0
Total Frames Received           = 1
Total Discarded TLVs            = 0
Total Unrecognized TLVs         = 0
Total Ageouts                   = 0
root@host (~)# lldptool -t -n -i eth2
Chassis ID TLV
    MAC: ec:13:db:41:63:00
Port ID TLV
    Local: 508
Time to Live TLV
    120
System Name TLV
    sw1
System Description TLV
    Juniper Networks, Inc. qfx5100-48s-6q Ethernet Switch, kernel JUNOS 13.2X51-D38, Build date: 2015-06-12 02:33:47 UTC Copyright (c) 1996-2015 Juniper Networks, Inc.
System Capabilities TLV
    System capabilities:  Bridge, Router
    Enabled capabilities: Bridge, Router
Port Description TLV
    xe-0/0/0
MAC/PHY Configuration Status TLV
    Auto-negotiation not supported and not enabled
    PMD auto-negotiation capabilities: 0x8000
    MAU type: Unknown [0x0000]
Link Aggregation TLV
    Aggregation capable
    Currently not aggregated
    Aggregated Port ID: 0
Maximum Frame Size TLV
    1514
Port VLAN ID TLV
    PVID: 1
Unidentified Org Specific TLV
    OUI: 0x009069, Subtype: 1, Info: 564633373136303530303437
VLAN Name TLV
    VID 1: Name vlan-1
LLDP-MED Capabilities TLV
    Device Type:  netcon
    Capabilities: LLDP-MED, Network Policy, Location Identification, Extended Power via MDI-PSE
End of LLDPDU TLV

Other helpful links:

http://comments.gmane.org/gmane.linux.network/408868 https://communities.intel.com/thread/87759 https://sourceforge.net/p/e1000/mailman/message/34129092/

And my Google search

Answer 2

It's a Firmware feature that can be toggled off

Since October 13, 2017, Intel released a version of their driver 2.3.6 that support toggling off the LLDP handling using a private-flag. This is done by executing the following command:

sudo ethtool --set-priv-flags <interface name> disable-fw-lldp on

• replace <interface name> with your interface name. (example - eth0)

Download Intel's Driver i40e for X710/ XL710 version 2.3.6

---

Installation Instructions (source)

1. Move the base driver tar file to the directory of your choice. For
   example, use '/home/username/i40e' or '/usr/local/src/i40e'.

2. Untar/unzip the archive, where <x.x.x> is the version number for the
   driver tar file:
   tar zxf i40e-<x.x.x>.tar.gz

3. Change to the driver src directory, where <x.x.x> is the version number
   for the driver tar:
   cd i40e-<x.x.x>/src/

4. Compile the driver module:
   # make install
   The binary will be installed as:
   /lib/modules/<KERNEL VERSION>/updates/drivers/net/ethernet/intel/i40e/i40e.ko

   The install location listed above is the default location. This may differ
   for various Linux distributions.
   NOTE:Â To compile the driver on some kernel/arch combinations, a
   package with the development version of libelf (e.g. libelf-dev,
   libelf-devel, elfutilsl-libelf-devel) may need to be installed.

  NOTE: To gather and display additional statistics, use the
  I40E_ADD_PROBES pre-processor macro:
  #make CFLAGS_EXTRA=-DI40E_ADD_PROBES
  Please note that this additional statistics gathering can impact
  performance.

5. Load the module using the modprobe command:
   modprobe <i40e> [parameter=port1_value,port2_value]

   Make sure that any older i40e drivers are removed from the kernel before
   loading the new module:
   rmmod i40e; modprobe i40e

6. Assign an IP address to the interface by entering the following,
   where ethX is the interface name that was shown in dmesg after modprobe:

   ip address add <IP_address>/<netmask bits> dev ethX

7. Verify that the interface works. Enter the following, where IP_address
   is the IP address for another machine on the same subnet as the interface
   that is being tested:
   ping <IP_address>

---

This is from Intel's commit:

From: Dave Ertman

Implement the private flag disable-fw-lldp for ethtool to disable the processing of LLDP packets by the FW. This will stop the FW from consuming LLDPDU and cause them to be sent up the stack.

The FW is also being configured to apply a default DCB configuration on link up.

Toggling the value of this flag will also cause a PF reset.

Disabling FW DCB will also disable DCBx.

Answer 3

Created an init script to do this on start up of a machine. Any pull requests appreciated.

If anyone knows how to tell the status of the embedded lldp agent it would be appreciated. This could be adapted for systemd with some better exit codes.

https://github.com/timhughes/i40e-lldp-agent/

Answer 4

As the ethtool toggle does not seems to be persistent across reboots we've setup following udev rule.

/etc/udev/rules.d/10-disable-fw-lldp.rules:

ACTION=="add", SUBSYSTEM=="net", ENV{INTERFACE}=="*", DRIVERS=="i40e", PROGRAM="/usr/sbin/ethtool --set-priv-flags $name disable-fw-lldp on"