4

We have a Linux (debian) server that has two NICs that are connected to the same switch. (We used to have a switch dedicated for traffic between our servers but when most of our servers moved to a colo facility that switch moved as well.) These NICs have different static IP addresses but about once a month arpwatch will send out a pair of flip flop messages as one of the IPs bounces from one interface to the other and back. What could be causing this?

5
  • 1
    You don't say anything about the OS or the hardware. It might be helpful if you did. Aug 24, 2009 at 21:28
  • On DHCP? I honestly can't think of a reason this would ever happen on fixed IPs.
    – Ernie
    Aug 24, 2009 at 21:36
  • 1
    Is the flip-flop timed around a reboot, by chance?
    – Bill Weiss
    Aug 24, 2009 at 21:37
  • I've edited the question to reflect that the server is running linux and uses static IP addresses. Aug 24, 2009 at 22:53
  • @Bill, We reboot this server about once a year, so no it's not related to a reboot. Aug 24, 2009 at 23:02

4 Answers 4

7

This might be related to Linux ARP behavior with multiple NICs on the same network and has been discussed in this Server Fault question. Basically you need to set some sysctl parameters:

net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2

These will cause ARP replies to only go out on the network interface that actually owns the IP address being requested in the ARP request.

As mentioned in the above-referenced question this is because by default in Linux the host owns an IP, and not a particular interface, so the replies will go out on whatever interface it chooses. This can cause problems when you have multiple interfaces on the same network and is exactly what the above sysctl parameters modify.

2

if it is on linux AND you are using udev (probably are, it's been "standard" for a few years now), you can force the assignment of eth0 or eth1 etc to specific MAC addresses.

e.g. i have the following in /etc/udev/rules.d/70-persistent-net.rules on an IBM x3655 fileserver at work. it has 4 NICs, three of them plugged in to 3 different VLANs, and the final one plugged into the "internal Gb backbone" (a dedicated Gigabit switch in the rack for all inter-server communications - NFS, ntp, dns, rsync, etc). Obviously, it is crucial that each interface gets the same name and the same IP address when it reboots:

# This file was automatically generated by the /lib/udev/write_net_rules
# program run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single line.

# built-in broadcom extreme NICs
## PCI device 0x14e4:0x164a (bnx2)
SUBSYSTEM=="net", ACTION=="add",DRIVERS=="?*", ATTR{address}=="00:14:5e:5a:18:ac", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add",DRIVERS=="?*", ATTR{address}=="00:14:5e:5a:18:ae", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI Intel e1000 NICs
## PCI device 0x8086:0x105e (e1000)
SUBSYSTEM=="net", DRIVERS=="?*", ATTR{address}=="00:15:17:2e:e0:c3", NAME="eth2"
SUBSYSTEM=="net", DRIVERS=="?*", ATTR{address}=="00:15:17:2e:e0:c2", NAME="eth3"
2
  • I looked at the equivalent file on this server and I noticed that two of the interfaces (but not the two that are flip floping) have the name "eth0". What could go wrong with that? Aug 24, 2009 at 23:00
  • well, lots could go wrong with that. or nothing. depends. are the two "eth0" entries both for NICs that are still currently installed in the machine? or is one or both of them for old NICs that have been replaced? either way, it would be a good idea to clean up that file so that it accurately reflects current reality.
    – cas
    Aug 24, 2009 at 23:43
1

I don't have an answer, but maybe a hint. A while ago a colleague of mine mentioned that there had been a bug in the linux driver for a particular network card. This bug would apparently lead to the MAC(!) addresses bouncing between interfaces if the server had more than one. I never checked this, but I seem to remember that this bug was supposedly fixed with 2.6.17 or thereabouts.

1

Just to clarify Kamil's solution for those unfamiliar with sysctl, edit this file:

vim /etc/sysctl.conf

Add these two lines:

net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2

and then run:

sudo sysctl -p

so that they take effect

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .