Docker: Isolate compose stacks on network level, while allowing a single service to be reverse proxied by Traefik

Question

I'm trying to secure containers on my homelab.

The main goal is:

• isolating nginxA and nginxB, so they are not able to talk to each other via 172.17.0.1 (eg. preventing nginxA from reaching nginxB via 172.17.0.1:5001)
• isolating nginxA and nginxB from localhost, making them exclusively accessible via traefik
• Allowing nginxA to talk to nginxA_DB, which are in the same stack, but without allowing other containers nor traefik to talk to nginxA_DB

Each docker stack has a nginx/apache service with it's port exposed in docker-compose in the following manner:

docker-compose-nginxA.yml:

networks:
  internal:
    ipam:
      config:
        - subnet: 10.0.0.0/29

...SNIP...

nginxA:
  networks:
    internal:
      ipv4_address: 10.0.0.2
  ports:
    - "172.17.0.1:5000:80"

docker-compose-nginxB.yml:

networks:
  internal:
    ipam:
      config:
        - subnet: 10.0.0.8/29

...SNIP...

nginxB:
  networks:
    internal:
      ipv4_address: 10.0.0.10
  ports:
    - "172.17.0.1:5001:80"

Making these services only accessible via localhost (172.17.0.1) or traefik,

Traefik being also on the same machine with the following configuration:

http:

  routers:

    nginxA:
     entryPoints:
       - web
     service: nginxA

    nginxB:
     entryPoints:
       - web
     service: nginxB


...SNIP...

  services:

    nginxA:
      loadBalancer:
        servers:
          - url: http://172.17.0.1:5000

    nginxB:
      loadBalancer:
        servers:
          - url: http://172.17.0.1:5001

An idea was to connect all the containers in a same /24 network with traefik, without exposing any ports via compose, but a single traefik network wouldn't isolate the containers from each other, only from the outside.

Another idea was to create a /31 network for each container and merge all the networks traefik, but I'm unsure if this will do the isolation as intended. eg.

nginxA:
  networks:
    internal:
      ipv4_address: 10.0.0.2 # /29
    nginxA-traefik:
      ipv4_address: 10.50.0.1 # /31

traefik:
  networks:
    nginxA-traefik:
    nginxB-traefik:
  ...

Thanks for reading this far! Do you have any ideas how this can be done?

Answer 1

You're on the right track. As in your last example, you want to create a separate network for each communication path you want between two containers. These can all be /31's or /29s or whatever you want as long as the subnets are different.

Example:

• nginxA-traefik connecting nginxA and traefik
• nginxB-traefik connecting nginxB and traefik
• nginxA-nginxA_DB connecting nginxA and nginxA_DB

You should not do any port mapping on the nginx containers since traefik will communicate with them directly over the docker networks. This will allow each nginx container to communicate with traefik (and vice versa) but they can't communicate with each other.

---

However! This does not scale very well because you'll have to create/manage one network for every service connected to traefik which will quickly get unwieldy. A better way would be to create a single network for all traefik-linked services (e.g. traefik_public), connect all of your service containers (i.e. nginx_A, nginx_B and anything else) to this network, and add some rules to iptables to only allow traffic to/from traefik.

I wrote a pretty simple container firewall that will take care of the iptables rules for you automatically: https://github.com/kaysond/trafficjam You just specify the network you want to protect (traefik_public) and the container to allow communication to/from (e.g. ancestor=traefik:latest) and it takes care of the rest!