2

One of our severs is behaving strangely with networking on startup. We've been using configuration changes and ifup/ifdown to try to isolate the issue, with frustratingly inconsistent results. We have a configuration that is consistent with other servers where we are not having this issue. A significant difference is that this server is from a different hardware vendor, but I don't know why that would be causing the problem we're having.

When we bring up the eth0 interface with sudo ifup eth0, the bonded interface bond0, and the vlan interface bond0.3001, are brought up, and appear correctly configured from ifconfig, but the default route is missing.

user@admin1:~$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.1.1.0        0.0.0.0         255.255.255.0   U     0      0        0 bond0.3001

Repeated attempts to bring the interface down (using sudo ifdown bond0, which seems to be the easiest way to bring down the whole multi-layered config with a single command) and then back up, don't seem to ever add the route correctly. However, if we edit the bond0.3001 entry in the interfaces file /etc/network/interfaces, it will frequently (but not always, argh) bring up the interfaces, and the correct default route.

user@admin1:~$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.1.1.1        0.0.0.0         UG    0      0        0 bond0.3001
10.1.1.0        0.0.0.0         255.255.255.0   U     0      0        0 bond0.3001

We have been commenting in/out the "network" line below as the file edit, with somewhat consistent results; when we make the edit, and bounce the interface stack, the default route is present pretty frequently. When we bounce the interface stack without making the edit, it seems to fail 100% of the time. Adding the default route manually with sudo route add default gw 10.1.1.1 bond0.3001 has worked every time we've tried, but we need the system to restart networking reliably.

Here is our config:

user@admin1:~$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual
    bond-master bond0
auto eth1
iface eth1 inet manual
    bond-master bond0

# The primary network interface
#Aggregate bond
auto bond0
iface bond0 inet manual
    bond-slaves none
    bond-mode 802.3ad
    bond-miimon 100
    bond-lacp-rate 1

auto bond0.3001
iface bond0.3001 inet static
    address 10.1.1.14
    gateway 10.1.1.1
    netmask 255.255.255.0
    #network 10.1.1.0
    broadcast 10.1.1.255
    #post-up route add default gw 10.1.1.1 bond0.3001
    #post-down route del default gw 10.1.1.1 bond0.3001

One additional datapoint - in running sudo strace -f sudo ifup eth0 to try to see what the differences are, it takes longer on a successful attempt (default route created) than a failed attempt. (default route missing) This may just be other downstream networks services running, that fail quickly when there's no default route.

EDIT: getting suspicious that kernel updates are the culprit, update that seemed to resolve the issue for this server was 3.19.0-25 to 3.19.0-43. We've seen similar behavior on other servers after kernel updates. More testing on specific kernel versions would probably be necessary to track down the root cause.

  • Just to be sure, have you checked the two nic's with ethtool that they support it? What is the output of sudo mii-tool or ethtool eth0 and ethtool eth1 – Diamond Jan 11 '16 at 22:17
  • @bangal thanks for the help, but the interface definitely supports a default route. It worked automatically sometimes, and worked everytime when added manually. – righdforsa Feb 16 '16 at 0:06
2

I just hit this issue with a rack of Ubuntu 14.04.3 servers this week, all upgraded to linux-generic-lts-vivid and running the 3.19.0-43 kernel. After flailing for a few days, we solved the problem by downgrading the kernel to 3.16.0-57 (we had the same problem with the 3.19.0-2x kernel that ships with the 14.04.3 amd64 server iso).

apt-get remove linux-generic-lts-vivid -y
apt-get autoremove
apt-get install linux-generic-lts-utopic -y
rm -f /boot/*3.19*
update-grub
reboot

As far as symptoms, default routes were non-deterministically added during boot. We could always fix it via ifdown bond0 && ifup p4p2 p5p2, but, like you, we needed networking to work reliably on boot.

| improve this answer | |
  • Ah, that may have been it, @bjcube. After fighting with this for a couple of days, it seemed to go away, but we weren't precise enough about tracking all of the changes made, to make sure if it was a kernel update, but we did a dist-upgrade, which upgraded the linux-generic kernel from 3.19.0-25 to 3.19.0-43. Giving you the upvote for a pretty good guess. – righdforsa Feb 16 '16 at 0:04
  • Latest kernel update from 3.13.0.63.71 (most of our 14.04 servers are running 3.13, not 3.19, which was installed on the OP server) to 4.4.0.47.34 broke every ubuntu virt server it was installed on. Somehow, booting to single user mode, disabling autostart for kvm nodes, and rebooting "fixed it", but obviously that's not operationally sustainable. Rolled back to 3.13. Definitely think this is pointing to some bugs in interface loading somewhere in the kernel. – righdforsa Nov 10 '16 at 18:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.