I have a few hosts connected to the same switch, which are all on the same subnet (10.0.0.0/16). Two of these hosts have faster network interfaces so I have connected them together, meaning these two machines now have a direct link with each other without going through a switch.
I now need to set up the routing such that when these two machines try to talk to each other, the packets go over this faster direct link in preference to the slower link via the switch.
The easiest way would be to configure the direct link to be on a different subnet, however then I will need to use different IPs or hostnames depending on which interface to use, and as I would like to be able to deploy standard configs to all machines (e.g. NFS mounts using hostnames) and not have to maintain custom IP overrides in
/etc/hosts, I feel this solution would be too easy to get a hostname wrong and send traffic over the wrong interface.
What I am looking for is a way to tell the two Linux machines that even though
eth0 handles 10.0.0.0/16, when you want to communicate with 10.0.0.5, even though it's in eth0's subnet, send the packets through
I tried adding a host routing rule with
route add -host 10.0.0.5 dev eth1 which does send the packet out on the correct interface, however it comes from the wrong IP address (the direct link's subnet rather than the original subnet.)
I guess the only way to fix this is to set the same IP address on both interfaces, but will this cause any problems? Can a machine correctly have the same IP on multiple NICs without causing problems? I'm assuming I'll need to set routing metrics properly so that the NIC connected to the switch is given priority (to avoid all traffic for the subnet being sent to the other host by mistake), but is there anything else I need to be aware of with this set up? Can it lead to any other issues or difficult-to-resolve problems?
Or is there a better, more robust way to achieve this?
EDIT: Here is the extra into requested by @A.B:
$ ip -br link lo UNKNOWN 00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP> eth0 UP ec:f4:xx:xx:xx:a4 <BROADCAST,MULTICAST,UP,LOWER_UP> eth1 UP ec:f4:xx:xx:xx:a5 <BROADCAST,MULTICAST,UP,LOWER_UP> $ ip -br address lo UNKNOWN 127.0.0.1/8 eth0 UP 10.0.0.4/16 eth1 UP 10.0.99.4/24 $ ip route default via 10.0.0.1 dev eth0 proto static 10.0.0.0/16 dev eth0 proto kernel scope link src 10.0.0.4 10.0.0.5 dev eth1 scope link 10.0.99.0/24 dev eth1 proto kernel scope link src 10.0.99.4
I set up the direct link (
eth1) on a separate subnet and then tried the
route command in my original post, and this is where things are now. It looks like perhaps I need to get the
src attribute set for my direct route.