Linux server has 2 active network interfaces:
IF:eth1 IP:192.168.1.1/24 MAC:11:11:11:11:11:11 (1GbE) IF:eth2 IP:192.168.1.2/24 MAC:22:22:22:22:22:22 (10GbE)
The idea is that the 10GbE interface (eth2) is the primary interface for communication with hosts on the network. I want to leave the second 1GbE interface (eth1) up as a failsafe. In the event that the 10GbE interface goes down: I'd still have an easy way in, can update DNS so hosts can connect, etc.
While observing interface statistics I noticed that all traffic was sending/receiving on the eth1 instead of eth2 despite the fact that all hosts on the network are addressing this interface. I confirmed DNS A record points to the IP of the correct interface. Additionally, I confirmed that addressing the interface by IP instead of FQDN produces the same result.
I cleared the ARP cache on my machine and pinged the eth1 interface by IP address. I inspect my ARP table and find the MAC address of eth1. I cleared the ARP cache again and pinged the eth2 interface by IP address. Again, I inspect my ARP table and find the MAC address of eth1 (not eth2).
If I bring down eth1, physically disconnect the interface, or put it on a different logical network - I get the expected behavior, traffic goes over my eth2 interface.
My question: Why does this happen? I am seeing some evidence that this is expected behavior on the linux kernel due to its "weak host model."
How can I keep both interfaces up, on the same network, and have them work in the way that I expect.