I have a few hosts connected to the same switch, which are all on the same subnet ( Two of these hosts have faster network interfaces so I have connected them together, meaning these two machines now have a direct link with each other without going through a switch.

I now need to set up the routing such that when these two machines try to talk to each other, the packets go over this faster direct link in preference to the slower link via the switch.

The easiest way would be to configure the direct link to be on a different subnet, however then I will need to use different IPs or hostnames depending on which interface to use, and as I would like to be able to deploy standard configs to all machines (e.g. NFS mounts using hostnames) and not have to maintain custom IP overrides in /etc/hosts, I feel this solution would be too easy to get a hostname wrong and send traffic over the wrong interface.

What I am looking for is a way to tell the two Linux machines that even though eth0 handles, when you want to communicate with, even though it's in eth0's subnet, send the packets through eth1 instead.

I tried adding a host routing rule with route add -host dev eth1 which does send the packet out on the correct interface, however it comes from the wrong IP address (the direct link's subnet rather than the original subnet.)

I guess the only way to fix this is to set the same IP address on both interfaces, but will this cause any problems? Can a machine correctly have the same IP on multiple NICs without causing problems? I'm assuming I'll need to set routing metrics properly so that the NIC connected to the switch is given priority (to avoid all traffic for the subnet being sent to the other host by mistake), but is there anything else I need to be aware of with this set up? Can it lead to any other issues or difficult-to-resolve problems?

Or is there a better, more robust way to achieve this?

EDIT: Here is the extra into requested by @A.B:

$ ip -br link
lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP> 
eth0             UP             ec:f4:xx:xx:xx:a4 <BROADCAST,MULTICAST,UP,LOWER_UP> 
eth1             UP             ec:f4:xx:xx:xx:a5 <BROADCAST,MULTICAST,UP,LOWER_UP> 

$ ip -br address
lo               UNKNOWN
eth0             UP   
eth1             UP   

$ ip route
default via dev eth0 proto static dev eth0 proto kernel scope link src dev eth1 scope link dev eth1 proto kernel scope link src

I set up the direct link (eth1) on a separate subnet and then tried the route command in my original post, and this is where things are now. It looks like perhaps I need to get the src attribute set for my direct route.

  • 1
    You are trying to do something that breaks a lot of rules. If it us doable it is likely a lot more work then a couple of entries in a hosts file. I expect having the same IP on 2 interfaces will break a lot of stuff (confused ARP table). You might be able to bodge something together using iptables nat on both devices but the cure is worse then the problem. – davidgo May 23 at 20:55
  • Could you anyway give more informations on these hosts with the current (not well working) configuration? Could you add the output of ip -br link; ip -br address; ip route. Note that on Linux route is obsolete, and some features require the use of ip route because route can't express them. – A.B May 23 at 21:24

As is part of this IP address should be avoided. Else there would be a conflict with the actual address, even on eth0 considering Linux is using the Weak Host Model and would answer by default to ARP requests for this IP address also on eth0 for, creating ARP conflicts. Don't use conflicting IP addresses.


ip route flush dev eth1
ip address flush dev eth1

The standard method with glue IP addresses

Let's choose two unrelated addresses to be used by the two hosts. They have to not clash with anything else in use on your network, but as they are point-to-point /32 addresses, anything can do, they won't be used as part of a LAN, but only as point-to-point/peer addresses. I'll arbitrarily use and Should later more than 2 of those hosts inherit a faster switch and are connected together using this separate switch, this can be slightly amended and having related IP addresses in the same block is then again easier.

configuration for host

# /32 : no route gets created (beside the hidden *local* routing table)
ip address add dev eth1 
# add the peer (point to point) route on the same link
ip route add dev eth1

Actually the two command above have a shortcut, you can replace both of them with this single command below:

ip address add peer dev eth1

Now tell the host that to reach (which is more specific than there's a route using the peer IP address, but preferring a different source IP address than what would be chosen by default (the obsolete route command can't do this):

ip route add via dev eth1 src

With this in place you get:

# ip route get via dev eth1 src uid 0 

There's one minor drawback: IP broadcasts are still sent to eth0 and if Strict Reverse Path Forwarding is active (either sysctl net.ipv4.conf.eth0.rp_filter or sysctl net.ipv4.conf.all.rp_filter gives 1 rather than 0 or 2) those broadcasts, when sent by the peer (eg running on peer host something similar to echo test | socat udp4-datagram:,broadcast -) will be ignored because received on the now wrong interface. So if you are using protocols relying on this and already apply a Strict Reverse Path Forwarding, switch eth0 to Loose mode if needed:

sysctl -w net.ipv4.conf.eth0.rp_filter=2

The equivalent configuration for host

ip address add peer dev eth1
ip route add via dev eth1 src
sysctl -w net.ipv4.conf.eth0.rp_filter=2

For example on Debian-like ifupdown interface configuration files you can use the pointopoint keyword and a few up additional commands for any command that doesn't have a direction configuration equivalent. (sysctl would rather be put in /etc/sysctl.d).

Simplified method without additional (nor duplicate) IP addresses

Actually the only role of and is to resolve link layer addresses to know the route for and there are used as some kind of glue that doesn't play any other role. Those IP addresses will be completely invisible, and no IP packet will ever use or in their content (except if explicitly using those), only ARP requests and answers will. There's no need to use such glue IP addresses at all.

For example the host provider Hetzner gives an example:

ip route add dev tap0

to reach an IP address on an interface without configuring an IP address on this interface (nor having this interface used as bridge port). In this example the peer on tap0 (which is a tun/tap device in Ethernet mode linked to a VM on the other side) has to answer ARP requests to resolve link-layer addresses.

But then again for symetrical reasons it doesn't need an IP address configured there either if it already configured it elsewhere, to answer properly an ARP request done through eth1: that's again part of Linux' implementation of the Weak Host Model.

So this can simply be used for host, without involving any extra IP address using only a single command:

ip route add dev eth1

Or to specify the source (to avoid ambiguity in case the host has more than one):

ip route add dev eth1 src

And for host

ip route add dev eth1 src

For accepting "slow" broadcasts on eth0 from the peer, they still require as before:

sysctl -w net.ipv4.conf.eth0.rp_filter=2

ARP requests to resolve their IP addresses can be answered on both interfaces (as linked above Linux does this by default), but here resolution or entries on the usual (old) side eth0 if any(eg: before those settings are put in place) won't trigger effects such as ARP flux because both peers are configured together to use eth1 leaving no other possible interpretation for the routes.

Choose what method you prefer. The first is more classical, the second has a simpler setup (but you might get a few "this can't work" from your peers). Remember that manually added routes are lost when an interface is administratively put down then up, so those settings must be put in an adequate network configuration setting to stay properly in effect.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.