I've two interfaces, let's say
eth0.4000 as vlan.
Both have a default-gateway.
Everything works as expected when a process listens on the interface directly.
But not for
hostPort bindings of Kubernetes.
vlan.gw-mac > eth0-mac, ethertype 802.1Q (0x8100), length 78: vlan 4000, p 0, ethertype IPv4 (0x0800), clientIP.38712 > vlanIP.80: Flags [S] eth0-mac > eth0.gw-mac, ethertype IPv4 (0x0800), length 74: vlanIP.80 > clientIP.38712: Flags [S.]
The SYN comes from
vlan.gateway, gets forwarded to the Container but the answer SYN-ACK leaves the stack through
eth0.gw and not the correct
vlan.gw but tcpdump shows that the sourceIP is
The routing tables looking good:
# ip route get to <clientIP> from <vlanIP> dev eth0.4000 <clientIP> from <vlanIP> via <vlan.gw> dev eth0.4000 table 1 uid 0
The hostPort mapping gets created through the CNI-Plugin portmap which uses DNAT and SNAT (details linked). So the gateway lookup happens to early. When I manually add a route from the container-ip to lookup table 1 it works using the vlan interface but breaks eth0.
So the questions is - what has to be done that the routing happens after NAT replaced the container-ip with the interface-ip?