I have a ubuntu 18 server with 4 active NICs. The setup is:
1. one public (not relevant for this case)
2. one single eno1 with ip 10.123.10.21
3. one bond0 (LACP bond consisting of eno3 and eno4), ip 10.123.10.30
2/eno1 & 3/bond0 are in the same network/vlan as bond0 is supposed to handle application traffic and eno1 is for mgmt and supporting services..
When enabling firewall (ufw) for some services on eno1 nothing seemed to work and after extensive digging and tcpdumping I figured out that bond0 is answering to arp requests for eno1 and all traffic goes to bond0 instead. And as long as the fw rules are not IP specific, things work but when you say allow to IP on IF nothing gets trough as this IP is actually being served by another IF.
I don't have ip forwarding on or any bridging
# cat /proc/sys/net/ipv4/ip_forward 0
On the switch side this VLAN is access only/untagged for internal traffic only, one LACP group for the bond0, the rest are standard. It is also ufw/iptables independent as I turned the service off and it stays the same (I thought it could be some pre-routing chain).
So I'm quite confused as to what's going on. I don't know what I could post to be useful. ip addr + routes seem ok:
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 94:18:82:7b:9a:6c brd ff:ff:ff:ff:ff:ff inet 10.123.10.21/24 brd 10.123.10.255 scope global eno1 valid_lft forever preferred_lft forever 4: eno3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000 link/ether 2a:08:3c:37:0d:3a brd ff:ff:ff:ff:ff:ff 5: eno4: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000 link/ether 2a:08:3c:37:0d:3a brd ff:ff:ff:ff:ff:ff 8: bond0: <BROADCAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000 link/ether 2a:08:3c:37:0d:3a brd ff:ff:ff:ff:ff:ff inet 10.123.10.30/24 brd 10.123.10.255 scope global bond0 valid_lft forever preferred_lft forever inet6 fe80::2808:3cff:fe37:d3a/64 scope link valid_lft forever preferred_lft forever
# ip r default via ... dev eno50 proto static 10.123.10.0/24 dev bond0 proto kernel scope link src 10.123.10.30 10.123.10.0/24 dev eno1 proto kernel scope link src 10.123.10.21 .../28 dev eno50 proto kernel scope link src ....
If anyone has any ideas on where to move on from here I'd be grateful..
As it turns out this is the default/preferred behavior of linux tcp stack. From a very helpful comment by @Zoredache and fiddling with various parameters it turns out if you want to restrict traffic on each NIC in the same subnet to that NIC's IP, you need to set up policy based routing as well.
Some helpful links from here on:
and the very in-depth Lin ux Advanced Routing
I think I'll just accept the world as it is and modify my ufw filters to match this philosophy.