0

I'm running docker (v 18) on Debian buster on a setup where the server has 2 NIC connected. I'v a bridge setup on each of the 2 NIC (vlan interfaces) and each bridge has an IP assigned within a /24:

br-trusted : vlan 11, 192.168.11.6/24
be-untrusted: vlan 10, 192.168.10.6/24

The firewall is routing 2 /26 toward the bridge's ip like this:

route 192.168.11.64/26 to 192.168.11.6
route 192.168.10.64/26 to 192.168.10.6

On the docker host, there are numerous docker networks created with /30 subnet assigned out of the /26 (type bridge).
There is also policy routing set up (ip rules on the from) on the docker host so that traffic coming from either of the /26 is routed to the firewall:
traffic from 192.168.11.64/26 is routed to 192.168.11.1
traffic from 192.168.10.64/26 is routed to 192.168.10.1

The docker host itself is not doing any NATing in this setup and ip forwarding is enabled. I don't think it has any influence but to be exhaustive, the host is setup with AppArmor.

Although the fw is allowing this traffic, connectivity between 192.168.10.64/26 and 192.168.11.64/26 does not work. I can see the begining of a tcp handshake on the firewall (first packets) being sent from the container under 192.168.10.0/26 but no reply to this.
Besides, connectivity to 192.168.11.64/26 is working while connecting from another subnet (ie: 192.168.12.0/24)

ip route output

root@server:~# ip route
default via 192.168.10.1 dev br-untrusted proto static metric 426 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.10.0/24 dev br-untrusted proto kernel scope link src 192.168.10.6 metric 426 
192.168.10.64/30 dev br-3763c83356f0 proto kernel scope link src 192.168.10.65 
192.168.10.72/30 dev br-f371a5a7d04a proto kernel scope link src 192.168.10.73 
192.168.10.76/30 dev br-7f353ded66a3 proto kernel scope link src 192.168.10.77 
192.168.10.192/26 via 192.168.10.5 dev br-untrusted proto static metric 426 
192.168.11.0/24 dev br-trusted proto kernel scope link src 192.168.11.6 metric 425 
192.168.11.64/30 dev br-041f7f27b0ae proto kernel scope link src 192.168.11.65 
192.168.11.68/30 dev br-d3f5ade8b635 proto kernel scope link src 192.168.11.69 linkdown 
192.168.11.72/30 dev br-ae989d4b5739 proto kernel scope link src 192.168.11.73 
192.168.11.76/30 dev br-5dd2902444c4 proto kernel scope link src 192.168.11.77 
192.168.11.80/30 dev br-9a41e27c8e5a proto kernel scope link src 192.168.11.81 
192.168.11.84/30 dev br-91f9d2b59f40 proto kernel scope link src 192.168.11.85 
192.168.11.88/30 dev br-f0de1756566c proto kernel scope link src 192.168.11.89 
192.168.11.92/30 dev br-a13be477ede5 proto kernel scope link src 192.168.11.93 

root@server:~# ip route show table trusted
default via 192.168.11.1 dev br-trusted 
192.168.11.0/24 dev br-trusted scope link 
root@server:~# 
root@server:~# ip route show table untrusted
default via 192.168.10.1 dev br-untrusted 
192.168.10.0/24 dev br-untrusted scope link 

ip rule output

root@server:~# ip rule
0:  from all lookup local 
32762:  from 192.168.10.64/26 lookup untrusted 
32763:  from 192.168.11.64/26 lookup trusted 
32764:  from 192.168.10.64/26 lookup untrusted 
32765:  from 192.168.11.64/26 lookup trusted 
32766:  from all lookup main 
32767:  from all lookup default 
root@server:~# 

script creating the routing/rules

#!/bin/sh

test -f /run/iproute.done && exit 0
touch /run/iproute.done

ip route add default via 192.168.11.1 dev br-trusted table trusted
ip route add 192.168.11.0/24 dev br-trusted table trusted
ip rule add from 192.168.11.64/26 table trusted

ip route add default via 192.168.10.1 dev br-untrusted table untrusted
ip route add 192.168.10.0/24 dev br-untrusted table untrusted
ip rule add from 192.168.10.64/26 table untrusted

command used to create the docker network

docker network create --ipv6 --subnet="192.168.10.64/30" --gateway="192.168.10.65" --ip-range="192.168.10.64/30" nw_nginx

Content of /etc/docker/daemon.json

{
    "storage-driver": "btrfs",
    "mtu": 0,
    "debug": false,
    "log-level": "",
    "selinux-enabled": false,
    "userns-remap": "",
    "group": "",
    "cgroup-parent": "",
    "default-ulimits": {},
    "ipv6": true,
        "fixed-cidr-v6": "ipv6_prefix:AAAA:AAAA:ff00::/104",
    "iptables": false,
    "bridge": "",
    "bip": "",
    "raw-logs": false,
    "registry-mirrors": [],
    "insecure-registries": [],
        "dns": ["192.168.11.66"],
        "dns-search": ["example.net"]
}
  • Some further testing, running a tshark while trying to connect from 192.168.10.66 to 192.168.11.94 show the TCP SYN packet getting in but no reply back. Same test but source ip from another subnet (192.168.12.13) shows a normal tcp connection. – Nicolas Apr 3 at 20:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.