I have a Linux system with 3 dedicated NICs & all 3 are on the same subnet with the following IPs
ETH0: 192.168.1.10
ETH1: 192.168.1.11
ETH2: 192.168.1.12
I've setup three routing tables & corresponding IP routes & rules to isolate the traffic of the 3 ports. Shown below is the same for ETH1. ETH0 & ETH2 have similar entries as well.
default via 192.168.1.1 dev eth1
192.168.1.0/24 dev eth1 src 192.168.1.11
32752: from all to 192.168.1.11/24 lookup 1
32753: from 192.168.1.11/24 lookup 1
If ETH1 & ETH2 are disconnected while ETH0 is connected, when I ping ETH1 & ETH2 IPs, I get a response from ETH0. In another question, Ghost ping on a multi-NIC Linux system I found out that this is caused by the ARP table being populated with the same MAC address for all 3 ports.
I've set arp_ignore & arp_announce as specified here Ubuntu Linux - multiple NICs, same LAN… ARP responses always go out a single NIC & I can see the ARP entries on the PC containing the correct MAC addresses.
However, I still see that only one port becomes the "default" in this circumstance. This can be seen very easily when say ETH1 & ETH2 are already connected and the cable for ETH0 is now plugged in but ETH0 is connected to an isolated switch (which has no connection to the subnet). Now, ETH0 becomes the default & due to the isolation, pings to ETH1 & ETH2 also fail.
Is there a way I can truly isolate the traffic on the three ports?