So I am experiencing some weirdness on one of my servers. It has two on board gigabit ethernet ports on it's motherboard both of which seem to have been detected correctly (Broadcom NetXtreme II Gigabit Ethernet). Both ports are plugged into two separate managed switches on the same subnet.
I have given both eth0 and eth1 independent IP addresses on the same subnet (10.0.10.20 and 10.0.10.30 respectively). I have verified that I can ping both of them.
On the managed switches if I turn off the port that eth1 is connected too I can no longer ping 10.0.10.30 as expected. Turning the port back on resume the service. When I turn off the port that eth0 is connected to I can no longer ping either address. Turning the port back on resumes the service for both addresses.
"ifdown eth0" also takes down both interfaces (though ifconfig shows eth1 as still up)
If I put eth0 on a separate network while still being physically up I encounter the same problem of both interfaces not responding (this is my biggest issue with this problem).
There are no errors generated in the boot log or messages log, and the messages log reflects that both interfaces go completely down whenever I take eth0 off the same network as eth1 for any reason.
The only errors that I have seen what-so-ever are on ifconfig where at the very bottom of eth0 and eth1 it will show one of three messages (they are not consistent and not always there). The messages are:
Googling those has resulted in some vague questions, but never a solution.