2

I have a VPS running multiple docker containers. I have a nextcloud instance, which get's SSL/TLS termination by an nginx proxy (certificates from Let'sEncrypt). And I have an openvpn container. In it's docker network I also host further services (own bind dns server and a git server), that I can reach trough the VPN.

Now I also want to reach my nextcloud instance through the VPN. Originally I thought this wouldn't be a problem, since the nextcloud instance can be reached through the internet and the VPN also gives connection to the internet. But unfortunately I cannot reach it. If I curl my server (http or https) through the VPN I get "port 80/443: No route to host". Without connecting the VPN the connection works correctly.

If I use traceroute, I can see, that it correctly reaches the public IP of my VPS. So I conclude, that it is a problem with the routing. The traffic targeted to port 80/443 on the public IP of my VPS don't get forwarded/routed to the nginx proxy container (which exposed the mentioned ports).

As I understood, docker uses firewalld/iptables to route the traffic between and to containers. Thus other rules are applied to the VPN traffic, than the traffic comming from the internet. What do I need to configure how, so that the VPN traffic (server internal) to my public IP address is forwarded correctly to the corresponding container? I would like to maintain constant/unchanged connectivity between VPN & No-VPN states, so that my Nextcloud app does not get confused.

What I have tried: I tried out possibilities of a workaround. I could add an own DNS entry for my nextcloud instance in my VPN DNS server, which points to the IP of the nextcloud app container (where I would loose SSL/TLS termination) or of the nginx proxy. In the last case the nginx proxy does not forward the traffic to the nextcloud container, since it uses a different hostname. I want to leave the proxy configuration unchanged, if possible, since it is automatically filled at container startup/from the letsencrypt companion container. Also the certificates would not match up with the used FQDN. If I try to add a master zone with my real/public DNS name (so that I can use the same FQDN as from the outside), all other domains from that TLD don't get forwarded anymore (Is there a possibility to configure bind for that?).

TL;DR: Traffic from a docker container to the public VPS IP address does not get forwarded to the correct docker container, like traffic from outside does.

If you also need more information about the used containers, I will add links and my docker-compose files.

EDIT:

[root@XXXXXXXX ~]# iptables -S FORWARD
-P FORWARD ACCEPT
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-7e5cecc96f4a -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-7e5cecc96f4a -j DOCKER
-A FORWARD -i br-7e5cecc96f4a ! -o br-7e5cecc96f4a -j ACCEPT
-A FORWARD -i br-7e5cecc96f4a -o br-7e5cecc96f4a -j ACCEPT
-A FORWARD -o br-fd56ce52983e -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-fd56ce52983e -j DOCKER
-A FORWARD -i br-fd56ce52983e ! -o br-fd56ce52983e -j ACCEPT
-A FORWARD -i br-fd56ce52983e -o br-fd56ce52983e -j ACCEPT
-A FORWARD -o br-f1ef60d84b48 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-f1ef60d84b48 -j DOCKER
-A FORWARD -i br-f1ef60d84b48 ! -o br-f1ef60d84b48 -j ACCEPT
-A FORWARD -i br-f1ef60d84b48 -o br-f1ef60d84b48 -j ACCEPT
-A FORWARD -o br-b396aa5a2d35 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-b396aa5a2d35 -j DOCKER
-A FORWARD -i br-b396aa5a2d35 ! -o br-b396aa5a2d35 -j ACCEPT
-A FORWARD -i br-b396aa5a2d35 -o br-b396aa5a2d35 -j ACCEPT
-A FORWARD -o br-83ac9a15401e -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-83ac9a15401e -j DOCKER
-A FORWARD -i br-83ac9a15401e ! -o br-83ac9a15401e -j ACCEPT
-A FORWARD -i br-83ac9a15401e -o br-83ac9a15401e -j ACCEPT
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct
-A FORWARD -j FORWARD_IN_ZONES_SOURCE
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD -j FORWARD_OUT_ZONES_SOURCE
-A FORWARD -j FORWARD_OUT_ZONES
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
8
  • Is the nginx proxy container running in the host network space? Or do connections from outside get routed to it through iptable rules? – Gerrit Feb 25 '20 at 17:25
  • I think the connections get routed to the nginx proxy, as I simply have set a network for the container in the docker compose file. I adapted this docker-compose file from the nextcloud docker repo on github. It set's the network proxy-tier for the proxy and opens it's ports 80 and 443. Unfortunately I'm not very firm in the networking details with docker. – chrisl Feb 25 '20 at 17:44
  • I have been doing some experiments. It seems docker uses a combination of a user program for proxying and iptables forwarding and nat'ing. If for some reason the iptables rules are deleted, the user program takes over the TCP connections from outside and from the main network space, but because it is not visible to the other containers, you can't connect from them. Can you look at the output of iptables -S FORWARD? There should be at least 6 rules there referring to docker and the docker bridge. – Gerrit Feb 26 '20 at 23:17
  • @Gerrit I added the output of the command. I see 4 forward rules for the docker0 interface and 4 forward rules for each of my defined docker bridge networks. The proxy-tier network is the one with the ID 7e5cecc96f4a – chrisl Feb 27 '20 at 8:49
  • Ok, I take it the openvpn container is on a bridge of its own. Traffic between different container bridges is not allowed by default. It's the DOCKER-ISOLATION-STAGE-1 chain you see in the FORWARD chain. If you remove it temporarily with iptables -D FORWARD -j DOCKER-ISOLATION-STAGE-1 you should be able to reach the nextcloud through the VPN. You can reinsert the isolation with iptables -I FORWARD 2 -j DOCKER-ISOLATION-STAGE-1. – Gerrit Feb 27 '20 at 12:36
2
+50

Docker by default does not allow traffic between any two of its containers that are connected to different bridges. And also it does not allow traffic from a container to a port that has been mapped to the outside by docker itself. This is all implemented with iptables.

First off, the mapping of a port to the outside also happens with iptables. It uses a DNAT rule in the nat table. For these rules Docker creates a separate DOCKER chain, so that the same rules apply from PREROUTING or OUTPUT in the nat table. The DNAT rules are preceded with RETURN jumps that filter out all traffic coming from a Docker bridge. So that is the first hurdle.

It looks a bit like this:

-A DOCKER -i br-one -j RETURN
-A DOCKER -i br-two -j RETURN
-A DOCKER ! -i br-one -p tcp -m tcp --dport EXPOSEDPORT -j DNAT --to-destination 172.17.0.2:INTERNALPORT

The DNAT rule can also have a -d address if you exposed the port to that local address only. No traffic from any Docker bridge can hit the DNAT rule(s) because of the RETURN rules before that. And also on top of that, the DNAT rule does not allow a DNAT back through the same bridge the traffic came from. Which wouldn't be necessary anyway, because from the same bridge you can just reach the INTERNALPORT already.

The restriction on traffic between containers on different bridges is implemented in the filter table of iptables. Two custom chains are at the beginning of the FORWARD chain, and the default policy of that chain is DROP. One is for containers with user-defined bridges, the other for containers with Docker bridges: DOCKER-ISOLATION-STAGE-1. That chain again uses DOCKER-ISOLATION-STAGE-2. The combination of both says basically that if traffic leaves a Docker bridge and then enters another Docker bridge, then DROP it (without ICMP signaling, so the connection just hangs.....)

It looks like this:

-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A DOCKER-ISOLATION-STAGE-1 -i br-one ! -o br-one -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-two ! -o br-two -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-2 -o br-one -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-two -j DROP

So, if you want traffic from bridge one, to hit a DNAT for a port exposed on the outside by a container on bridge two and you want the traffic to return for a full connection, then you have to do a couple of things:

  • Drop the RETURN rules that stops the traffic from DNAT in the DOCKER chain in the nat table. You HAVE to remove the RETURN for the source bridge. You CAN leave the RETURN for the destination bridge, if you don't want to allow a container from that bridge to access a DNAT exposed port.

    • iptables -t nat -D DOCKER -i br-one -j RETURN
    • iptables -t nat -D DOCKER -i br-two -j RETURN #Optional if br-one -> br-two
  • Remove the DROP rules for both bridges from the DOCKER-ISOLATION-STAGE-2 chain in the filter table.

    • iptables -t filter -D DOCKER-ISOLATION-STAGE-2 -o br-one -j DROP
    • iptables -t filter -D DOCKER-ISOLATION-STAGE-2 -o br-two -j DROP

Now the lines are open.

Docker does not often refresh its rules (at least not in the 19.03 version I tested with). It seems it only rebuilds the rule sets when the docker daemon restarts, not when you stop or start or create a container. You could try to tack any changes to the service restart to keep them persistent.

3
  • I will try that tomorrow. Though I hoped, that I could implement this inside of docker, which would make a much cleaner solution, that I could configure in the docker-compose files. Is there no way to do this in docker? I think I have already tried to add the proxy and the VPN server to the same docker network, but I'm currently not sure. Will also try that tomorrow. If both are at the same bridge network, wouldn't that mean, that the traffic wouldn't be dropped? – chrisl Feb 27 '20 at 20:06
  • I do not have much experience at all with docker compose. I do know that if you include the proxy and the VPN in the same bridge, that you CAN reach the proxy from the VPN server, but only on the internal address and port of the proxy, not on the exposed port and address pair. – Gerrit Feb 27 '20 at 20:40
  • I can confirm this to work. As I understand, with this I break down the isolation of docker networks for all networks, that should be able to communicate with my public IP address. Thank you for you work – chrisl Mar 1 '20 at 11:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.