1

I have a ubuntu 18 server with 4 active NICs. The setup is:
1. one public (not relevant for this case)
2. one single eno1 with ip 10.123.10.21
3. one bond0 (LACP bond consisting of eno3 and eno4), ip 10.123.10.30

2/eno1 & 3/bond0 are in the same network/vlan as bond0 is supposed to handle application traffic and eno1 is for mgmt and supporting services..

When enabling firewall (ufw) for some services on eno1 nothing seemed to work and after extensive digging and tcpdumping I figured out that bond0 is answering to arp requests for eno1 and all traffic goes to bond0 instead. And as long as the fw rules are not IP specific, things work but when you say allow to IP on IF nothing gets trough as this IP is actually being served by another IF.

I don't have ip forwarding on or any bridging

# cat /proc/sys/net/ipv4/ip_forward
0

On the switch side this VLAN is access only/untagged for internal traffic only, one LACP group for the bond0, the rest are standard. It is also ufw/iptables independent as I turned the service off and it stays the same (I thought it could be some pre-routing chain).

So I'm quite confused as to what's going on. I don't know what I could post to be useful. ip addr + routes seem ok:

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 94:18:82:7b:9a:6c brd ff:ff:ff:ff:ff:ff
    inet 10.123.10.21/24 brd 10.123.10.255 scope global eno1
       valid_lft forever preferred_lft forever
4: eno3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 2a:08:3c:37:0d:3a brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 2a:08:3c:37:0d:3a brd ff:ff:ff:ff:ff:ff
8: bond0: <BROADCAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 2a:08:3c:37:0d:3a brd ff:ff:ff:ff:ff:ff
    inet 10.123.10.30/24 brd 10.123.10.255 scope global bond0
       valid_lft forever preferred_lft forever
    inet6 fe80::2808:3cff:fe37:d3a/64 scope link 
       valid_lft forever preferred_lft forever
# ip r
default via ... dev eno50 proto static 
10.123.10.0/24 dev bond0 proto kernel scope link src 10.123.10.30 
10.123.10.0/24 dev eno1 proto kernel scope link src 10.123.10.21 
.../28 dev eno50 proto kernel scope link src .... 

If anyone has any ideas on where to move on from here I'd be grateful..

SOLUTION

As it turns out this is the default/preferred behavior of linux tcp stack. From a very helpful comment by @Zoredache and fiddling with various parameters it turns out if you want to restrict traffic on each NIC in the same subnet to that NIC's IP, you need to set up policy based routing as well.

Some helpful links from here on:
https://pontus.ullgren.com/view/Multiple_interfaces_on_the_same_subnet
and the very in-depth Lin ux Advanced Routing
https://lartc.org/lartc.html

I think I'll just accept the world as it is and modify my ufw filters to match this philosophy.

Jure

2

Not certain, but I you need to do something with arp_filter, and/or maybe some of the other arp flags as well.

https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt

arp_filter - BOOLEAN

...

0 - (default) The kernel can respond to arp requests with addresses from other interfaces. This may seem wrong but it usually makes sense, because it increases the chance of successful communication. IP addresses are owned by the complete host on Linux, not by particular interfaces. Only for more complex setups like load-balancing, does this behaviour cause problems.

  • Hmm, good pointer. Don't seem to have that parameter in /proc/net/ipv4 and sysctl doesn't seem to accept setting it, but I'll look at the rest, perhaps something will do the trick. I didn't know about the host-owns-IPs concept, NICs seem a more proper place but there must be a reason. If nothing else, it helps with on NIC dying I guess.. – Jure May 15 at 18:32
  • Ok, this settings are present but in /proc/net/conf/*/ as they can be set per interface. For starters, for IF not to answer an arp request arp_ignore=1 seems to make an IF stop answering to other IPs. Actually it seems you need to enable policy based routing as well.. – Jure May 15 at 19:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.