10

Note: I already have a workaround for this problem (as described below) so this is only a "want-to-know" question.

I have a productive setup with around 50 hosts including blades running xen 4 and equallogics providing iscsi. All xen dom0s are almost plain Debian 5. The setup includes several bridges on every dom0 to support xen bridged networking. In total there are between 5 and 12 bridges on each dom0 servicing one vlan each. None of the hosts has routing enabled.

At one point in time we moved one of the machines to a new hardware including a raid controller and so we installed an upstream 3.0.22/x86_64 kernel with xen patches. All other machines run debian xen-dom0-kernel.

Since then we noticed on all hosts in the setup the following errors every ~2 minutes:

[55888.881994] __ratelimit: 908 callbacks suppressed
[55888.882221] Neighbour table overflow.
[55888.882476] Neighbour table overflow.
[55888.882732] Neighbour table overflow.
[55888.883050] Neighbour table overflow.
[55888.883307] Neighbour table overflow.
[55888.883562] Neighbour table overflow.
[55888.883859] Neighbour table overflow.
[55888.884118] Neighbour table overflow.
[55888.884373] Neighbour table overflow.
[55888.884666] Neighbour table overflow.

The arp table (arp -n) never showed more than around 20 entries on every machine. We tried the obvious tweaks and raised the

/proc/sys/net/ipv4/neigh/default/gc_thresh*

values. FInally to 16384 entries but no effect. Not even the interval of ~2 minutes changed which lead me to the conclusion that this is totally unrelated. tcpdump showed no uncommon ipv4 traffic on any interface. The only interesting finding from tcpdump were ipv6 packets bursting in like:

14:33:13.137668 IP6 fe80::216:3eff:fe1d:9d01 > ff02::1:ff1d:9d01: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff1d:9d01, length 24
14:33:13.138061 IP6 fe80::216:3eff:fe1d:a8c1 > ff02::1:ff1d:a8c1: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff1d:a8c1, length 24
14:33:13.138619 IP6 fe80::216:3eff:fe1d:bf81 > ff02::1:ff1d:bf81: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff1d:bf81, length 24
14:33:13.138974 IP6 fe80::216:3eff:fe1d:eb41 > ff02::1:ff1d:eb41: HBH ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff1d:eb41, length 24

which placed the idea in my mind that the problem maybe related to ipv6, since we have no ipv6 services in this setup.

The only other hint was the coincidence of the host upgrade with the beginning of the problems. I powered down the host in question and the errors were gone. Then I subsequently took down the bridges on the host and when i took down (ifconfig down) one particularly bridge:

br-vlan2159 Link encap:Ethernet  HWaddr 00:26:b9:fb:16:2c  
          inet6 addr: fe80::226:b9ff:fefb:162c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:120 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:5286 (5.1 KiB)  TX bytes:726 (726.0 B)

eth0.2159 Link encap:Ethernet  HWaddr 00:26:b9:fb:16:2c  
          inet6 addr: fe80::226:b9ff:fefb:162c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1801 errors:0 dropped:0 overruns:0 frame:0
          TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:126228 (123.2 KiB)  TX bytes:1464 (1.4 KiB)

bridge name bridge id       STP enabled interfaces
...
br-vlan2158     8000.0026b9fb162c   no      eth0.2158
br-vlan2159     8000.0026b9fb162c   no      eth0.2159

The errors went away again. As you can see the bridge holds no ipv4 address and it's only member is eth0.2159 so no traffic should cross it. Bridge and interface .2159 / .2157 / .2158 which are in all aspects identical apart from the vlan they are connected to had no effect when taken down. Now I disabled ipv6 on the entire host via sysctl net.ipv6.conf.all.disable_ipv6 and rebooted. After this even with bridge br-vlan2159 enabled no errors occur.

Any ideas are welcome.

2 Answers 2

5

I believe your problem is because of a kernel bug that was patched in net-next.

Multicast snooping gets disabled when the bridge is initialized because of a bug trying to rehash the table. IGMP snooping stops the bridge from forwarding every HBH ICMPv6 multicast query reply, which results in the neighbour table filling up with ff02:: neighbours from multicast replies which it should not see (try ip -6 neigh show nud all).

The proper workaround is to attempt to re-enable snooping like: echo 1 > /sys/class/net/eth0/bridge/multicast_snooping. The alternative is to make the neighbour table gc thresholds bigger than the number of hosts in the broadcast domain.

The patch is here.

1
  • I had to do echo 1 > /sys/class/net/br0/bridge/multicast_snooping. Jul 8, 2013 at 7:06
3

what's the return of ip route show cache table all when you're experiencing this error ?

arp -n or ip neigh show will only show some of the entries in the cache.

ip route show cache table all will be much more detailed (and will include a lot of v6 related entries).

We tried the obvious tweaks and raised the /proc/sys/net/ipv4/neigh/default/gc_thresh*

Did you do the same for ipv6 ? that solved the problem for us

Bye,

-- creis

1
  • 1
    ip route show cache table all did not reveal much more entries. I fixed the error messages by setting net.ipv6.neigh.default.gc_thresh1 = 1024 net.ipv6.neigh.default.gc_thresh2 = 2048 net.ipv6.neigh.default.gc_thresh3 = 4096) through sysctl.
    – tim
    Apr 7, 2012 at 12:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .