3

Over the last couple of days, the bonded network interface on one of our servers has stopped responding.

Looking through the kernel logs, I notice when the interface goes down, we are getting lots of repeated errors of the form:

[76019.645601] e1000e 0000:03:00.0 p9p1: speed changed to 0 for port p9p1
[76325.575540] e1000e 0000:03:00.0 p10p1: speed changed to 0 for port p10p1

Having had a quick search around for similar issues, I haven't been able to find anyone having reported this sort of behaviour before.

To provide a few more details on the server's configuration:

  • Both of the bonded network interfaces are associated to Intel 82574L ethernet controllers.
  • The server is running Ubuntu 16.04, with Linux kernel version 4.4.0-101-generic.
  • The bonded network interface has the following configuration:

    auto p9p1
    iface p9p1 inet manual
    bond-master bond0
    
    auto p10p1
    iface p10p1 inet manual
    bond-master bond0
    
    auto bond0
    iface bond0 inet static
    address 10.0.0.10
    gateway 10.0.0.1
    netmask 255.255.255.0
    bond-mode 4
    bond-miimon 100
    bond-lacp-rate 1
    bond-slaves p9p1 p10p1
    dns-nameservers 10.0.0.2 10.0.0.3
    
  • When the network interface goes down restarting the networking service on the server, by running service networking restart, seems to remedy the issues

I was wondering if anyone had experienced similar issues before and or has any suggestions for debugging the cause of something like this?

5
  • Have you performed some package firmware/update? Some driver updates require also newer firmware and newer firmware may require newer drivers... Nov 24, 2017 at 12:22
  • @JaroslavKucera I haven't done so explicitly but I guess something like this could have been done for me through the Ubuntu system package upgrades. Nov 24, 2017 at 12:24
  • 1
    You can search apt logs for that (the most likely it's the kernel driver). Then I'd suggest to update the firmware of the NIC to the latest. It may be also some bug in the kernel NIC or bond driver. In that case try to use previous kernel version. Nov 24, 2017 at 12:28
  • I've yet to confirm this but I suspect this issue is possibly related to the following bug in the Ubuntu linux package bug tracker: bugs.launchpad.net/ubuntu/+source/linux/+bug/1730550 Nov 24, 2017 at 15:14
  • If it confirms, please make it answer. Nov 24, 2017 at 15:16

2 Answers 2

1

It would appear that, for me, these issues were likely caused by a known bug in the Linux kernel v4.4.0-97-generic on Ubuntu 16.04: e1000e in 4.4.0-97-generic breaks 82574L under heavy load.

Having applied the patched test kernel version v4.4.0-98, submitted by the bug's assignee, on the Ubuntu Linux package bug tracker I have not since experienced the erroneous behaviour returning after a weekend of fairly heavy load testing of the bonded interface.

1

I just hit the same error messages.. but in my case issue wasn't on the server side at all. The stderr prints not only e1000e NIC, but all 4 of them. With cable disconnect/connect the messages are reproduced. So different drivers have same behavior... and after we did software debug on server, then cabling (changing cables with new ones), what's rest was the top of rack switch.

A switch reboot solved it.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .