1

I have a machine connected to a slightly broken scientific instrument and I'm trying to work around the problems. The link is point to point, 2x25G ethernet, no other devices involved. The device streams UDP data with one stream per interface and I can configure the destination IP and MAC address, but the source IP address for the device is the same for both interfaces. Using tcpdump I can see that the traffic is arriving on the right interfaces with the correct destination IP and mac address for both.

But I can only listen on one of the IPs and get the data, if I listen on the other IP there is no traffic. The kernel is obviously dropping the traffic and I don't know why, nor how to debug this.

The machine is running ubuntu 18.04, IPs 10.50.5.1/16 and 10.50.5.129/16 the device has IP 10.50.1.10. Routes:

10.50.0.0/16 dev data0 proto kernel scope link src 10.50.5.1 
10.50.0.0/16 dev data1 proto kernel scope link src 10.50.5.129

Any Suggestions?

I can see in netstat that the packets aren't being dropped, so where do they go?

Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
data0     9600   202320      0      0 0            29      0      0      0 BMRU
data1     9600   203600      0      0 0            31      0      0      0 BMRU
  • Is there a firewall enabled? iptables -L -v -n – Mark Wagner Feb 14 at 18:24
1

You have routing issues due to trying to reach the same destination using two different interfaces and routes. By default routing is done per destination, not also per source. By default Linux, which uses the weak host model (but Ubuntu also sets reverse path filter, see later) will choose the same interface: the one among equal candidates appearing first in its list. So here, both source IP addresses 10.50.5.1 and 10.50.5.129 will be using data0 to reach 10.50.1.10 because it's currently the first among the two equals in the route entries.

To do otherwise requires source based routing (aka policy routing) on Linux: have the source also be used to determine the route to the destination.

Also for similar reasons, in your current configuration: having two IP addresses in the same IP LAN (10.50.0.0/16) Linux will answer ARP queries in the LAN from any of its interfaces with any of its MAC addresses. In the end, unless like in your setup with a device using permanent ARP entries rather than doing requests, usually only one interface gets to receive all incoming traffic (it might even be the "wrong" one).

And the last straw is that even with the instrument sending to the correct interface with the correct IP address destination, Ubuntu's default is to have Strict Reverse Path enabled, dropping packets received on the "wrong" interface. That's the reason you miss data: the interface (try the modern ip -statistics link command) received them, but the routing stack dropped them right after. To verify this, just ask the kernel's routing stack what it thinks about sending and receiving IP packets:

  • sending

    # ip route get from 10.50.5.1 10.50.1.10
    10.50.1.10 from 10.50.5.1 dev data0 uid 0 
        cache 
    # ip route get from 10.50.5.129 10.50.1.10
    10.50.1.10 from 10.50.5.129 dev data0 uid 0 
        cache 
    

    Both would be sent using data0, ignoring data1.

  • receiving

    # ip route get from 10.50.1.10 iif data0 10.50.5.1
    local 10.50.5.1 from 10.50.1.10 dev lo table local 
        cache <local> iif data0 
    # ip route get from 10.50.1.10 iif data1 10.50.5.129
    RTNETLINK answers: Invalid cross-device link
    

    Receiving on data1 is forbidden by strict reverse path filter.

You could choose to disable reverse path, but if you ever need to send on a specific interface that won't help anyway.

I propose a cleaner method below.


First, to avoid having to mingle with ARP settings (and "ARP flux" considerations), consider the two interfaces for what they are used as: point-to-point IP interfaces, even if used over Ethernet link layer. There's thus no need to use a whole /16 LAN: just point-to-point routes will do.

# ip address flush dev data0
# ip address flush dev data1

# ip address add 10.50.5.1 peer 10.50.1.10/32 dev data0
# ip address add 10.50.5.129 peer 10.50.1.10/32 dev data1

The two commands above are mostly shortcuts for ip address add 10.50.5.N/32 dev dataX; ip route add 10.50.1.10/32 dev dataX.

You could have chosen to keep your /16 network settings (eg: if finally they get used for real LAN usage with a farm of equipments behind), but this would have required to also toggle arp_filter=1 on the interfaces to stay on the safe side.

To solve the source based routing issue: use rules pointing to additional routing tables, which will have a partial copy of some routes from the main routing table. I choose arbitrary values 1000 and 1001 for those routing tables (and use arbitrary fixed rule priorities 10000 10001 even if not required). Everything about table 1000 is not really needed (because it's the default), but it's just cleaner anyway.

ip route add table 1000 10.50.1.10/32 dev data0 src 10.50.5.1
ip route add table 1001 10.50.1.10/32 dev data1 src 10.50.5.129

ip rule add pref 10000 from 10.50.5.1 lookup 1000
ip rule add pref 10001 from 10.50.5.129 lookup 1001

The main table will still determine what interface is used by default (probably data0): any tool not binding to a specific IP address will use this (and thus 10.50.5.1) as default, same if binding to 10.50.5.1. But now any tool binding to 10.50.5.129 will have bidirectional traffic properly using data1.

# ip route get 10.50.1.10
10.50.1.10 dev data0 src 10.50.5.1 uid 0 
    cache 
# ip route get from 10.50.5.1 10.50.1.10
10.50.1.10 from 10.50.5.1 dev data0 table 1000 uid 0 
    cache 
# ip route get from 10.50.5.129 10.50.1.10
10.50.1.10 from 10.50.5.129 dev data1 table 1001 uid 0 
    cache 

and now, incoming data on data1 is not dropped by reverse path filter:

# ip route get from 10.50.1.10 iif data0 10.50.5.1
local 10.50.5.1 from 10.50.1.10 dev lo table local 
    cache <local> iif data0 
# ip route get from 10.50.1.10 iif data1 10.50.5.129
local 10.50.5.129 from 10.50.1.10 dev lo table local 
    cache <local> iif data1 

Note:

The additional routing tables are populated manually, not by the kernel. If an interface is brought down (or an IP is removed), then brought back up (or the IP added back) the corresponding table 1000 or 1001 will be flushed and will not be re-populated: it has to be added back manually (or with a network configuration tool able to do this).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.