0

I have a Linux system with 3 NICs & all 3 are on the same subnet. I've setup policy routing on the system with ip rules & ip routes using 3 tables. These are the IPs of the 3 NICs.

ETH0: 192.168.1.10

ETH1: 192.168.1.11

ETH2: 192.168.1.12

One thing I've observed is that, if the cables to ETH1 & ETH2 are disconnected, a ping to their IPs from a PC connected to ETH0 via a switch can sometimes be successful.

I understand that this is a consequence of having the IPs on the same subnet (although, shouldn't this be avoided when I've used IP rules?) but what I fail to understand is how the switch can successfully route the packets destined for the IP belonging to ETH1 or ETH2 to the IP that's meant for ETH0? In other words, how can a packet destined for 192.169.1.12 be sent to 192.168.1.10 by the switch? Doesn't its internal routing table have an entry only for 192.168.1.10 & not for the other 2 IPs cause their cables are disconnected?

Improve this question
1
  • The accepted answer puts for the correct idea & explains the problem. If anyone looking at this question is in the need of a solution that can be implemented practically, see the top answer to this question - serverfault.com/questions/22253/…
    – Sunny Yates
    Mar 15, 2018 at 3:04

1 Answer 1

Reset to default
2

It's not only about routing (layer 3) but also about ARP (linking layer 2 and 3). Linux will typically answer all ARP requests with the same MAC, that means peers will have only one MAC address in their ARP cache for the 3 IPs and will reach only one card. It's possible that sometimes the elected MAC will change, causing all kind of troubles.

There's an available feature to prevent this. That's the arp_filter setting, but it also requires a few routing rules (per ip and/or per interface). Using only arp_filter or only routing tables might give worse results than not using them at all. Both are needed together. Because OP didn't show what routing policies were used, I put a complete solution below, considering no special routing policies were in place. Values for routing tables (100, 101, 102) were chosen at random.

  • LAN

    • try before:

      ip route get 192.168.1.1 from 192.168.1.10
ip route get 192.168.1.1 from 192.168.1.11
ip route get 192.168.1.1 from 192.168.1.12

      Note the results: all are using the same card (that doesn't mean it's always what happens anyway).
      On a peer server in the same LAN, as fast as possible:

      ping -c1 192.168.1.10 &
ping -c1 192.168.1.11 &
ping -c1 192.168.1.12
ip neigh show to 192.168.1.10
ip neigh show to 192.168.1.11
ip neigh show to 192.168.1.12

      most certainly all 3 IPs resolve to the same MAC, thus card: the only card that will receive traffic from this peer.

    • alter the settings (Warning: risk of connectivity loss):

      echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_filter
echo 1 > /proc/sys/net/ipv4/conf/eth1/arp_filter
echo 1 > /proc/sys/net/ipv4/conf/eth2/arp_filter

ip route add 192.168.1.0/24 dev eth0 table 100
ip route add 192.168.1.0/24 dev eth1 table 101
ip route add 192.168.1.0/24 dev eth2 table 102

ip rule add from 192.168.1.10 lookup 100
ip rule add from 192.168.1.11 lookup 101
ip rule add from 192.168.1.12 lookup 102

    • flush peers' ARP caches (with DAD requests doubling as GARP) to speed up recovery if needed. Not needed at boot time since there were no such caches. Use iputils' arping, not the standalone arping tool (RHEL: iputils, Debian: iputils-arping, not the arping package), because syntax differs.

      arping -c5 -s 192.168.1.10 -D 192.168.1.10 -I eth0 &
arping -c5 -s 192.168.1.11 -D 192.168.1.11 -I eth1 &
arping -c5 -s 192.168.1.12 -D 192.168.1.12 -I eth2 &

    • redo the previous checks and compare: everything should be as expected, and neighbour tables on peer servers should show one different MAC per IP, matching the correct card's MAC.

  • WAN

    The default route will still use the "default" settings from table main, most likely using eth0. This route has to be duplicated on each table. Assuming the default router is 192.168.1.1 (else, adjust) that would be:

    ip route add default via 192.168.1.1 dev eth0 table 100
ip route add default via 192.168.1.1 dev eth1 table 101
ip route add default via 192.168.1.1 dev eth2 table 102

Now if a cable is disconnected, the expected result will happen: the IP set on the corresponding NIC will become unavailable 100% of the time.

Improve this answer
6
  • Thanks! This shed quite some light on the topic. However, there's still a problem. Without the arping, the peer will continue to retain the old entries & even when you reboot the host, the peer's contents aren't changed. In fact, if you clear the peer's table manually & then ping each port like you've mentioned, the MAC addresses that're populated belong to only one of the 3 ports & aren't unique as expected.
    – Sunny Yates
    Mar 14, 2018 at 4:19
  • Ok let's say arping is mandatory. Does this work as expected after a reboot ONCE you re apply again all the commands I put in this answer?
    – A.B
    Mar 14, 2018 at 7:56
  • If an arping is done, then yes, it does work. However, just setting the arp_filter at boot time doesn't do the job. The ARP table on the peer receives the same MAC address as eth0 for all 3 IPs. I had to set the arp_ignore & arp_announce as mentioned here :-serverfault.com/questions/22253/… Note that I had to DISABLE arp_filter for this to work cause the 3 IPs were on the same network as well as subnet.
    – Sunny Yates
    Mar 14, 2018 at 9:14
  • Don't get me wrong! Your answer definitely proves the concept & puts forth the right idea but it doesn't work out as a solution that can be implemented practically. I'll accept the answer & put a comment into my question with a link to the other question.
    – Sunny Yates
    Mar 15, 2018 at 3:02
  • Ok I read the top solution and admit it's way simplier. BUT incomplete. I tested it and it could only work if I set at least rp_filter to loose mode: for your case: for i in eth0 eth1 eth2; do echo 2 > /proc/sys/net/ipv4/conf/$i/rp_filter; done (or just set all/rp_filter to 2 ). If that's really the case you should write your solution and set it at the accepted solution. If you won't I think I'll upate later my solution and add the alternate simpler method. I'd rather find exacly how the simpler method works and why it appear to also require relaxing the reverse path filtering
    – A.B
    Mar 15, 2018 at 7:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .