I am trying to reach my Nextcloud (fpm behind nginx) instance through a wireguard VPN, both running in a docker container.
My docker-compose.yml looks like:
version: "3.9"
services:
nextcloud:
image: nextcloud
container_name: nextcloud
restart: always
environment:
POSTGRES_DB: nextcloud
POSTGRES_USER: postgres
POSTGRES_PASSWORD: xxxxxxxxx
POSTGRES_HOST: postgres
NEXTCLOUD_DATA_DIR: /data
volumes:
- /mnt/data/nextcloud:/var/www/html
- /mnt/data/files:/data
depends_on:
- postgres
web:
image: nginx:latest
container_name: web
restart: always
volumes:
- /mnt/data/nextcloud:/var/www/html:ro
- /srv/nextcloud/nginx.conf:/etc/nginx/nginx.conf
expose:
- "80"
ports:
- "80:80"
depends_on:
- nextcloud
wireguard:
image: docker.myownregistry.xy/wireguard:latest
container_name: wireguard
restart: "unless-stopped"
environment:
- PUID=1000
- PGID=1000
- TZ=Europe/Berlin
- PORT_FORWARDS=web:80
volumes:
- /srv/wireguard/wg0.conf:/config/wg0.conf
cap_add:
- NET_ADMIN
- SYS_MODULE
sysctls:
- net.ipv4.ip_forward=1
depends_on:
- web
The iptables look like:
# docker exec -it wireguard iptables -t nat -L -n -v
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DNAT tcp -- wg0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 to:DOCKER_WEB_IP
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
6 424 DOCKER_OUTPUT all -- * * 0.0.0.0/0 127.0.0.11
Chain POSTROUTING (policy ACCEPT 2 packets, 160 bytes)
pkts bytes target prot opt in out source destination
6 424 DOCKER_POSTROUTING all -- * * 0.0.0.0/0 127.0.0.11
0 0 SNAT tcp -- * eth0 0.0.0.0/0 DOCKER_WEB_IP tcp dpt:80 to:DOCKER_WIREGUARD_IP
Chain DOCKER_OUTPUT (1 references)
pkts bytes target prot opt in out source destination
0 0 DNAT tcp -- * * 0.0.0.0/0 127.0.0.11 tcp dpt:53 to:127.0.0.11:41071
6 424 DNAT udp -- * * 0.0.0.0/0 127.0.0.11 udp dpt:53 to:127.0.0.11:39020
Chain DOCKER_POSTROUTING (1 references)
pkts bytes target prot opt in out source destination
0 0 SNAT tcp -- * * 127.0.0.11 0.0.0.0/0 tcp spt:41071 to::53
0 0 SNAT udp -- * * 127.0.0.11 0.0.0.0/0 udp spt:39020 to::53
# docker exec -it wireguard iptables -L -n -v
Chain INPUT (policy ACCEPT 4 packets, 381 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT tcp -- wg0 eth0 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 flags:0x17/0x02 ctstate NEW
0 0 ACCEPT all -- eth0 wg0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- wg0 eth0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
Chain OUTPUT (policy ACCEPT 7 packets, 561 bytes)
pkts bytes target prot opt in out source destination
First of all I can ping the wireguard peer from the VPN and get responses. When I send a HTTP request to the peer IP the request appears in nginx' log and the http client get's stuck waiting for a response.
A TCP dump shows that the HTTP GET request get's actually forwarded, but the server response doesn't (it retries several times):
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wg0, link-type RAW (Raw IP), capture size 262144 bytes
13:10:05.319590 IP WG_CLIENT_IP.39952 > WG_PEER_IP.80: Flags [S], seq 2013357617, win 64860, options [mss 1380,sackOK,TS val 625358504 ecr 0,nop,wscale 7], length 0
13:10:05.319809 IP WG_PEER_IP.80 > WG_CLIENT_IP.39952: Flags [S.], seq 2024405621, ack 2013357618, win 28960, options [mss 1460,sackOK,TS val 4067727362 ecr 625358504,nop,wscale 7], length 0
13:10:05.337819 IP WG_CLIENT_IP.39952 > WG_PEER_IP.80: Flags [.], ack 1, win 507, options [nop,nop,TS val 625358524 ecr 4067727362], length 0
13:10:05.337939 IP WG_CLIENT_IP.39952 > WG_PEER_IP.80: Flags [P.], seq 1:75, ack 1, win 507, options [nop,nop,TS val 625358525 ecr 4067727362], length 74: HTTP: GET / HTTP/1.1
13:10:05.338016 IP WG_PEER_IP.80 > WG_CLIENT_IP.39952: Flags [.], ack 75, win 227, options [nop,nop,TS val 4067727380 ecr 625358525], length 0
13:10:05.601611 IP WG_PEER_IP.80 > WG_CLIENT_IP.39952: Flags [P.], seq 1:1413, ack 75, win 227, options [nop,nop,TS val 4067727644 ecr 625358525], length 1412: HTTP: HTTP/1.1 302 Found
13:10:05.621031 IP WG_CLIENT_IP.39952 > WG_PEER_IP.80: Flags [.], ack 1, win 507, options [nop,nop,TS val 625358808 ecr 4067727380,nop,nop,sack 1 {1369:1413}], length 0
13:10:05.655547 IP WG_PEER_IP.80 > WG_CLIENT_IP.39952: Flags [.], seq 1:1369, ack 75, win 227, options [nop,nop,TS val 4067727697 ecr 625358808], length 1368: HTTP: HTTP/1.1 302 Found
13:10:05.885450 IP WG_PEER_IP.80 > WG_CLIENT_IP.39952: Flags [.], seq 1:1369, ack 75, win 227, options [nop,nop,TS val 4067727928 ecr 625358808], length 1368: HTTP: HTTP/1.1 302 Found
13:10:06.335429 IP WG_PEER_IP.80 > WG_CLIENT_IP.39952: Flags [.], seq 1:1369, ack 75, win 227, options [nop,nop,TS val 4067728378 ecr 625358808], length 1368: HTTP: HTTP/1.1 302 Found
13:10:07.245449 IP WG_PEER_IP.80 > WG_CLIENT_IP.39952: Flags [.], seq 1:1369, ack 75, win 227, options [nop,nop,TS val 4067729288 ecr 625358808], length 1368: HTTP: HTTP/1.1 302 Found
13:10:09.086287 IP WG_PEER_IP.80 > WG_CLIENT_IP.39952: Flags [.], seq 1:1369, ack 75, win 227, options [nop,nop,TS val 4067731129 ecr 625358808], length 1368: HTTP: HTTP/1.1 302 Found
13:10:12.685537 IP WG_PEER_IP.80 > WG_CLIENT_IP.39952: Flags [.], seq 1:1369, ack 75, win 227, options [nop,nop,TS val 4067734731 ecr 625358808], length 1368: HTTP: HTTP/1.1 302 Found
13:10:20.365456 IP WG_PEER_IP.80 > WG_CLIENT_IP.39952: Flags [.], seq 1:1369, ack 75, win 227, options [nop,nop,TS val 4067742416 ecr 625358808], length 1368: HTTP: HTTP/1.1 302 Found
13:10:35.086342 IP WG_PEER_IP.80 > WG_CLIENT_IP.39952: Flags [.], seq 1:1369, ack 75, win 227, options [nop,nop,TS val 4067757152 ecr 625358808], length 1368: HTTP: HTTP/1.1 302 Found
...
The weird thing is when I run a simple http echo server (e.g.: ealen/echo-server which uses nodejs express) instead I can reach it through the VPN without any complications.
It also didn't work when I was using the nextcloud container with a bundled apache server.
EDIT:
Here is the configuration of my host iptables
# iptables -L -n -v
Chain INPUT (policy ACCEPT 423K packets, 53M bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
5142K 18G DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
5142K 18G DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
3473K 5987M ACCEPT all -- * br-50f634c9c605 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
3955 238K DOCKER all -- * br-50f634c9c605 0.0.0.0/0 0.0.0.0/0
1665K 12G ACCEPT all -- br-50f634c9c605 !br-50f634c9c605 0.0.0.0/0 0.0.0.0/0
3891 234K ACCEPT all -- br-50f634c9c605 br-50f634c9c605 0.0.0.0/0 0.0.0.0/0
0 0 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 4
Chain OUTPUT (policy ACCEPT 153K packets, 97M bytes)
pkts bytes target prot opt in out source destination
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
12 824 ACCEPT tcp -- !br-50f634c9c605 br-50f634c9c605 0.0.0.0/0 172.18.0.3 tcp dpt:2049
19 1140 ACCEPT tcp -- !br-50f634c9c605 br-50f634c9c605 0.0.0.0/0 172.18.0.5 tcp dpt:80
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
0 0 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
1665K 12G DOCKER-ISOLATION-STAGE-2 all -- br-50f634c9c605 !br-50f634c9c605 0.0.0.0/0 0.0.0.0/0
5142K 18G RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (2 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 DROP all -- * br-50f634c9c605 0.0.0.0/0 0.0.0.0/0
1665K 12G RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
5142K 18G RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
# iptables -t nat -L -n -v
Chain PREROUTING (policy ACCEPT 67238 packets, 8661K bytes)
pkts bytes target prot opt in out source destination
74 4480 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 66720 packets, 8628K bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 4761 packets, 360K bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 5305 packets, 393K bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0
2 352 MASQUERADE all -- * !br-50f634c9c605 172.18.0.0/16 0.0.0.0/0
0 0 MASQUERADE tcp -- * * 172.18.0.3 172.18.0.3 tcp dpt:2049
0 0 MASQUERADE tcp -- * * 172.18.0.5 172.18.0.5 tcp dpt:80
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
0 0 RETURN all -- br-50f634c9c605 * 0.0.0.0/0 0.0.0.0/0
12 824 DNAT tcp -- !br-50f634c9c605 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:2049 to:172.18.0.3:2049
19 1140 DNAT tcp -- !br-50f634c9c605 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 to:172.18.0.5:80
