Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

So I am experiencing some weirdness on one of my servers. It has two on board gigabit ethernet ports on it's motherboard both of which seem to have been detected correctly (Broadcom NetXtreme II Gigabit Ethernet). Both ports are plugged into two separate managed switches on the same subnet.

I have given both eth0 and eth1 independent IP addresses on the same subnet (10.0.10.20 and 10.0.10.30 respectively). I have verified that I can ping both of them.

On the managed switches if I turn off the port that eth1 is connected too I can no longer ping 10.0.10.30 as expected. Turning the port back on resume the service. When I turn off the port that eth0 is connected to I can no longer ping either address. Turning the port back on resumes the service for both addresses.

"ifdown eth0" also takes down both interfaces (though ifconfig shows eth1 as still up)

If I put eth0 on a separate network while still being physically up I encounter the same problem of both interfaces not responding (this is my biggest issue with this problem).

There are no errors generated in the boot log or messages log, and the messages log reflects that both interfaces go completely down whenever I take eth0 off the same network as eth1 for any reason.

The only errors that I have seen what-so-ever are on ifconfig where at the very bottom of eth0 and eth1 it will show one of three messages (they are not consistent and not always there). The messages are:

Interrupt:185 Memory:ea000000-ea012800

Interrupt:17

Interrupt:18

Googling those has resulted in some vague questions, but never a solution.

share|improve this question
    
Is there anything strange in their BIOS configuration? You actually have one NIC with two ports, and it basically shows up as two in the OS. It can be configured to act as one, which is half what this sounds like; might also need a firmware upgrade. –  Chris S Oct 5 '10 at 13:27
    
I will go look into this, /var/log/messages does show that they have different PCI addresses (PCI Interrupt 0000:04:00.0[A] and PCI Interrupt 0000:06:00.0[A]). I will check none-the-less and for any firmware updates. –  TrueDuality Oct 5 '10 at 13:35
    
sigh so there are firmware updates available but they need to be run in Windows (the downloaded exe is a zip archive but full of dlls and exes :( ). There wasn't anything in the BIOS about the network cards beyond enabling/disabling them and whether to allow PXE booting... –  TrueDuality Oct 5 '10 at 13:51
    
So after installing windows, performing the firmware updates and installing CentOS back on to the server the problem persists. –  TrueDuality Oct 5 '10 at 18:52

2 Answers 2

Looking at this it strikes me as an arp flux problem, not a firmware issue. This can arise when you have two nics in the same subnet, and in your case on the same L2 topology. http://linux-ip.net/html/ether-arp.html

You can validate this by putting them on different ip/subnets and see if you still have the same issues. If you need to get this to work take a look at arp_filter and your routing very carefully.

share|improve this answer
    
If you read the question carefully you'll note this line: "If I put eth0 on a separate network while still being physically up I encounter the same problem of both interfaces not responding (this is my biggest issue with this problem)." This is referring to a separate physical network, I'm sorry I wasn't more clear on that –  TrueDuality Oct 5 '10 at 14:40
    
Are you keeping these with the same IP addressing? (On the same subnet?) –  Christopher Karel Oct 5 '10 at 14:47
    
Right, Christoper gets to the root of it. The routing table looks weird, I mean which interface is traffic suppose to traverse? The idea that the traffic will egress out of the interface it came in on is not necessarily correct. –  MattyB Oct 5 '10 at 14:55
    
No, when I switch to a different physical network I was setting an IP within our public IP space. The issue is not with gateways, all of the traffic is being tested from the respective local subnet. –  TrueDuality Oct 5 '10 at 17:39
    
So the two ips are on different subnets? Sorry its not quite clear. –  MattyB Oct 5 '10 at 19:45

You know, Linux doesn't really like two NICs on the same subnet. At least, not with the basic ifconfig tools. Only one of them gets used to transmit, they seem to share the same MAC, etc. This causes all sorts of weird things to happen. Just like MattyB just mentioned.

The best solution is to use the iproute2 tools (slightly easier to read blog post) to configure your NICs. Try this:

Initialize interfaces:

ip link set eth0 up
ip link set eth1 up

Configure IP addresses

ip addr add 10.0.10.20/24 dev ethe0
ip addr add 10.0.10.30/24 dev eth0

Then setting up the routing table so that one of those NICs is the 'default'. ip route add 10.0.0.0/16 via 10.0.10.20


--Christopher Karel

share|improve this answer
    
They do have two separate MAC addresses, and doesn't mind having X number of interfaces on the same subnet as long as a gateway is set. Even without a gateway set IPs on the local subnet will still be able to ping the addresses. The physical links are going down in this case. –  TrueDuality Oct 5 '10 at 17:37

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.