5

I have a server running Wireguard (thus needing masquerade) and a container running on port 2525.

I have the following iptables rules:

iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 25 -j DNAT --to-destination 172.18.0.1:2525
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

When connecting to server:2525 directly, the Docker container is able to see my real IP address (1.2.3.4). When connecting to port server:25, the Docker container is seeing the local IP that's provided by docker network:

Apr 07 12:45:46 mx postfix/smtpd[87]: lost connection after CONNECT from unknown[172.18.0.1]
Apr 07 12:45:46 mx postfix/smtpd[87]: disconnect from unknown[172.18.0.1] commands=0/0

How do I make sure the Docker container is properly seeing the public IP address while connecting to port 25 (and not only when connecting to port 2525).

Thanks

# iptables -L -n -v -t nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
52300 3131K DNAT       tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            tcp dpt:25 to:172.18.0.1:2525
 150K 8524K DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    2   120 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
 3385  256K MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0
1733K  104M MASQUERADE  all  --  *      !br-b147ffdbc9f3  172.18.0.0/16        0.0.0.0/0
    0     0 MASQUERADE  tcp  --  *      *       172.17.0.2           172.17.0.2           tcp dpt:53
    0     0 MASQUERADE  udp  --  *      *       172.17.0.2           172.17.0.2           udp dpt:53
    0     0 MASQUERADE  tcp  --  *      *       172.18.0.2           172.18.0.2           tcp dpt:25

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
   12  1419 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  br-b147ffdbc9f3 *       0.0.0.0/0            0.0.0.0/0
   56  3192 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5354 to:172.17.0.2:53
    0     0 DNAT       udp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:5354 to:172.17.0.2:53
  107  6020 DNAT       tcp  --  !br-b147ffdbc9f3 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:2525 to:172.18.0.2:25
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 32:d0:56:15:0a:64 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    altname ens3
    inet 159.223.80.86/20 brd 159.223.95.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.15.0.19/16 brd 10.15.255.255 scope global eth0:1
       valid_lft forever preferred_lft forever
    inet6 2400:6180:0:d0::f57:6001/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::30d0:56ff:fe15:a64/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 32:dc:4a:e4:27:be brd ff:ff:ff:ff:ff:ff
    altname enp0s4
    altname ens4
    inet 10.130.244.15/16 brd 10.130.255.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::30dc:4aff:fee4:27be/64 scope link
       valid_lft forever preferred_lft forever
4: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none
    inet 10.200.200.52/24 scope global wg0
       valid_lft forever preferred_lft forever
5: wg1: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none
    inet 10.222.111.1/24 scope global wg1
       valid_lft forever preferred_lft forever
6: br-b147ffdbc9f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:46:21:70:c0 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-b147ffdbc9f3
       valid_lft forever preferred_lft forever
    inet6 fe80::42:46ff:fe21:70c0/64 scope link
       valid_lft forever preferred_lft forever
7: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:66:22:41:91 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:66ff:fe22:4191/64 scope link
       valid_lft forever preferred_lft forever
9: veth31eff9d@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
    link/ether e6:fb:80:5d:c7:a3 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::e4fb:80ff:fe5d:c7a3/64 scope link
       valid_lft forever preferred_lft forever
19: veth01269f5@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-b147ffdbc9f3 state UP group default
    link/ether 36:f4:e7:43:5f:da brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::34f4:e7ff:fe43:5fda/64 scope link
       valid_lft forever preferred_lft forever
8
  • it looks like you have an SNAT postrouting rule when going to the docker interface, that you could delete. Can you please add the result of iptables -L -n -v -t nat ? Apr 11, 2022 at 18:30
  • 1
    Can you ellaborate more on your network infra/setup including the wireguard part? It's unclear for me (at this moment) why masquerade is necessary.
    – eKKiM
    Apr 12, 2022 at 14:01
  • @setenforce1 added to the initial post
    – Tuinslak
    Apr 13, 2022 at 14:46
  • @eKKiM I believe it's needed to properly route (i.e. it acts as a gateway/VPN: client -> WG server -> internet)
    – Tuinslak
    Apr 13, 2022 at 14:46
  • Is the WG server also your default gateway in your network? Can you add the output of ifconfig -a or ip addr?
    – eKKiM
    Apr 13, 2022 at 14:54

1 Answer 1

2
+50

Just let Docker handle the redirection, which is dynamic and could change when containers are added, removed or restarted. But see UPDATE below.

This redirection should not be to 172.18.0.1 which is the host and not the container. When the host receives such connection, it is handled by docker-proxy which proxies it to the container, losing the source IP address in the process.

Docker already DNAT + routes this port correctly (except from the host itself, where docker-proxy plays this role) in the very last rule of the ruleset, to the running container with the 172.18.0.2 address. Except it's configured to use port 2525 rather than port 25.

  107  6020 DNAT       tcp  --  !br-b147ffdbc9f3 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:2525 to:172.18.0.2:25

This should be fixed with Docker settings, not manual iptables rules that can't change whenever container layout changes. As port 25 is privileged, if Docker is running rootless, additional settings are needed, check the documentation about exposing privileged ports.


UPDATE (factoring in OP's comments): OP can't currently use -p 25:25 because docker-proxy clashes with the local host's SMTP server and competes for listening on port 25 on the host. That's the reason the initial (wrong) iptables redirection was made by OP.

One can either:

  • disable globally docker-proxy by running dockerd with the property userland-proxy set to false

    either as a parameter --userland-proxy=false or as a property "userland-proxy": false added to /etc/docker/daemon.json.

    This will then allow to use docker run ... -p 25:25 ... (as documented) without clash: the host will reach itself when reaching localhost or $HOSTNAME, remote systems will reach the container and no "Address already in use" will make the host's SMTP daemon or Docker's container fail at start.

  • or else add a manual redirection (with lengthy setup below to do it almost automatically)

    Whenever the container is restarted, there is a risk that its (internal) IP address will change. So this must be computed. So with a container named mx using a network named mx and a single IP address involved this can be done as explained below.

    Create a separate prerouting chain (so it can be flushed without having to flush anything else) and call it first:

    iptables -t nat -N mynat
    iptables -t nat -I PREROUTING -j mynat
    

    The container's IP address can be retrieved programmatically (for the simple case of a container named mx with a single address):

    containerip=$(docker container inspect --format '{{.NetworkSettings.IPAddress}}' mx)
    

    (or one could use jq: containerip=$(docker container inspect mx | jq '.[].NetworkSettings.IPAddress')

    Finding the bridge interface name is more convoluted, or at least I couldn't find a way for it using only docker ... inspect. So find its IP address on the host and query the host using ip address to find only the bridge interface where this specific IP address is set (requires the jq command.)

     bridgeip=$(docker network inspect --format '{{(index .IPAM.Config 0).Gateway}}' mx)
     bridgeinterface=$(ip -json address show to "$bridgeip"/32 | jq -r '.[].ifname')
    

    Flush and repopulate mynat each time the container is (re)started:

    iptables -t nat -F mynat
    iptables -t nat -A mynat ! -i "$bridgeinterface" -p tcp --dport 25 -j DNAT --to-destination "$containerip":25
    

    which would be for the current case:

    iptables -t nat -I mynat ! -i br-b147ffdbc9f3 -p tcp --dport 25 -j DNAT --to-destination 172.18.0.2:25

    And to be sure Docker's own firewalling rules don't block such traffic, do something similar starting in filter/FORWARD from the DOCKER-USER chain.

    Initially (if from boot, you might have to also create first DOCKER-USER):

    iptables -N myforward
    iptables -I DOCKER-USER 1 -j myforward
    

    Then later each time the container is (re)started:

    iptables -F myforward
    iptables -A myforward -p tcp ! -i "$bridgeinterface" -d "$containerip" -p tcp --dport 25 -j ACCEPT
    

    which would be for the current case:

    iptables -A myforward -p tcp ! -i br-b147ffdbc9f3 -d 172.18.0.2 -p tcp --dport 25 -j ACCEPT
    

Notes:

  • To simplify rules above and avoid some of the calculations, the container and its bridge network can be started with fixed IP addresses. See for example this SO Q/A: Assign static IP to Docker container.

  • Here's also an UL SE Q/A with an answer of mine about problems when interacting with Docker (it's geared for nftables but some parts about the DOCKER-USER chain or br_netfilter bridge interactions are still of interest): nftables whitelisting docker

12
  • Thanks. This is using docker run: docker run -d -P --network mx -p 2525:25 -v /srv/mx/:/mx/ --restart always --name mx mx
    – Tuinslak
    Apr 14, 2022 at 9:52
  • I'm running on port 2525 because, if I don't run a postfix on localhost, mail command doesn't work and email is not delivered. pastebin.com/RR2mDx9T (when running the container on port 25)
    – Tuinslak
    Apr 14, 2022 at 9:58
  • afaik docker userland proxy will literally proxy the connection thus the source ip will get changed..
    – eKKiM
    Apr 14, 2022 at 10:21
  • @eKKiM indeed, that's what I cover at docker-proxy.
    – A.B
    Apr 14, 2022 at 12:15
  • @Tuinslak so with this context (can't use port 25) the Docker rule already in place (the very last rule seen in the ruleset displayed in the question put in place by -p 2525:25) already handles it. Why are you then adding a rule overriding it?
    – A.B
    Apr 14, 2022 at 12:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.