0

I've successfully configured multiple network interfaces on the same subnet following this guide by RedHat: https://access.redhat.com/solutions/30564

I can ping from any NIC to hosts on the subnet, and all NICs can be pinged from hosts on the subnet. YAY! But establishing a TCP connection does not work anymore, due to missing return paths I assume.

Before I started, the routing table looked like this:

# ip route list
default via 10.10.10.254 dev bond0 proto static
default via 10.10.10.254 dev eno2 proto dhcp src 10.10.11.201 metric 100
default via 10.10.10.254 dev eno3 proto dhcp src 10.10.10.41 metric 100
default via 10.10.10.254 dev eno4 proto dhcp src 10.10.10.40 metric 100
default via 10.10.10.254 dev eno1 proto dhcp src 10.10.11.200 metric 100
10.10.10.0/23 dev bond0 proto kernel scope link src 10.10.10.3
10.10.10.0/23 dev eno2 proto kernel scope link src 10.10.11.201
10.10.10.0/23 dev eno3 proto kernel scope link src 10.10.10.41
10.10.10.0/23 dev eno4 proto kernel scope link src 10.10.10.40
10.10.10.0/23 dev eno1 proto kernel scope link src 10.10.11.200
10.10.10.254 dev eno2 proto dhcp scope link src 10.10.11.201 metric 100
10.10.10.254 dev eno3 proto dhcp scope link src 10.10.10.41 metric 100
10.10.10.254 dev eno4 proto dhcp scope link src 10.10.10.40 metric 100
10.10.10.254 dev eno1 proto dhcp scope link src 10.10.11.200 metric 100

Now, there are routing tables eno1...4 and bond0 with their respective routing rules:

# ip route list table eno1
default via 10.10.11.254 dev eno1
10.10.10.0/23 dev eno1 scope link src 10.10.11.200

And there are policy rules that define outbound paths for each interface:

# ip rule list
0:      from all lookup local
32761:  from 10.10.10.3 lookup bond0
32762:  from 10.10.10.40 lookup eno4
32763:  from 10.10.10.41 lookup eno3
32764:  from 10.10.11.201 lookup eno2
32765:  from 10.10.11.200 lookup eno1

Only after flushing the main routing table, I was able to ping from + to all interfaces.

After testing what's working, I discovered that I can only connect to a service (bound to 10.10.10.40:8080 on eno4) when I add back this route:

10.10.10.0/23 dev eno4 proto kernel scope link src 10.10.10.40

There's two things here which I don't understand:

  • Why does the kernel need this rule at all?

Why are the defined policy from 10.10.10.40 lookup eno4 and route 10.10.10.0/23 dev eno4 scope link src 10.10.10.40 not sufficient? I modified the route in table eno4 to include proto kernel without success, so the policy rule from 10.10.10.40 does not apply here? Why?

  • Why were there multiple routes to the same subnet when I started hacking, but now I can't add more than one?

Adding the original routes via ip route add only works for 1 route to the subnet, the other routes are rejected with: RTNETLINK answers: File exists That (kind of) makes sense to me, if the kernel doesn't use a round-robin algorithm to balance traffic over the different routes to the same subnet. But why were there 5 routes to the subnet to begin with?

Additional Info:

OS: Ubuntu 18.04 (Server) Network init is done by Cloud-Init + Systemd-Networkd

Edit: Feel free to link in some resources that explain linux routing terminology and concepts in the comments :)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.