We have the following scenario, a docker host with two NICs in two subnets,
ens8: IP 192.168.100.74/24 and
ens9: IP 172.20.102.24/25, the default bridge
docker0 IP 172.17.0.1/16 and a second bridge
app_net IP 172.21.0.1/24.
192.168.100.1 is the default gateway on host (internet gateway).
The docker container have a fixed outgoing IP using SNAT, see Four ways to connect a docker container to a local network - Using NAT. This is done by two commands, assign target address to host interface
ip addr add 192.168.100.234/24 dev ens8 and add iptables rule
iptables -t nat -I POSTROUTING -s 172.21.0.4 -j SNAT --to-source 192.168.100.234. Containers are created using IPs from bridge app_net.
On the host I can either ping/traceroute IPs in subnet
192.168.100.x and in
172.20.102.x and internet e.g. google.com. Within a container, I can ping/traceroute IPs in subnet
192.168.100.x, I can also ping e.g. google.com, but I cannot ping anything in
172.20.102.24. See the following output.
# traceroute -I 192.168.100.10 traceroute to 192.168.100.10 (192.168.100.10), 30 hops max, 46 byte packets 1 172.21.0.1 (172.21.0.1) 0.004 ms 0.003 ms 0.002 ms 2 192.168.100.10 (192.168.100.10) 0.180 ms 0.158 ms 0.028 ms # traceroute -I google.com traceroute to google.com (126.96.36.199), 30 hops max, 46 byte packets 1 172.21.0.1 (172.21.0.1) 0.004 ms 0.013 ms 0.002 ms 2 fritz.box (192.168.100.1) 2.144 ms 3.114 ms 2.697 ms 3 server.provider.net (xx.yy.250.5) 6.947 ms 7.042 ms 4.305 ms 4 etc... # traceroute -I 172.20.102.24 traceroute to 172.20.102.24 (172.20.102.24), 30 hops max, 46 byte packets 1 172.20.102.24 (172.20.102.24) 0.004 ms 0.003 ms 0.002 ms # traceroute -I 172.20.102.9 traceroute to 172.20.102.9 (172.20.102.9), 30 hops max, 46 byte packets 1 172.21.0.1 (172.21.0.1) 0.004 ms 0.003 ms 0.002 ms 2 * * * 3 * * * (etc. no response)
If I do not add the iptables SNAT rule, containers can ping/traceroute everything, also machines on subnet
I found an article about multiple NICs and docker(#) with a stripped down solution on serverFault, but neither this, nor a deep look into advanced docker networking(#) gave me a clue, why I cannot reach other machines in the ens9 subnet, except the network card itself from within a docker container.
Am I missing something, or is this network scenario just wrong?
(#) Not allowed to post more than two links because of reputation, but the serverFault link together with Google "Docker over multiple network interfaces" should help to find the links.