iptables traversal when connecting to docker client

Question

How can the docker-host reach an exported port without hitting the iptables DNAT-rule?

Bonus: Why are the iptables log messages only shown with curl 127.0.0.1:8082 but not with curl localhost:8082?

UPDATE: Similar to https://stackoverflow.com/questions/29629620, connecting to the container works with empty iptables rules too (# systemctl stop iptables clears all tables, sets policies to accept).

Setup

docker-compose.yml

version: '3.2'

services:
    webapp:
        image: httpd:latest
        networks:
            - webapp
        ports:
            - "8082:80"


networks:
    webapp:
        driver: bridge
#port mapping does not work for internal networks
#        internal: true

Observed Behaviour

1. Packet starts in OUTPUT chain
2. Packet leaves (lo), arrives again (lo)
3. Accepted by conntrack target in ^*filterINPUT chain

Expected Behaviour

$ curl 127.0.0.1:8082

Although the nat-table [is traversed only once per connection][1], the counters for the DNAT should not be zero; Traversal for first packet:

1. Packet starts in OUTPUT chain
2. Packet leaves unchanged (-> lo), arrives again (lo)
3. Packet still unchanged: ^*natPREROUTING -jump-> ^*natDOCKER
4. DNAT --to-destination 172.18.0.2:80
5. Routing decision --> ^*filterFORWARD (destination is foreign host)
6. ^*filterFORWARD -jump-> ^*filterDOCKER ang get accepted, further packets already accepted by conntrack target

Counters

*nat (UPDATE: full output instead of DOCKER-chain)
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 44 packets, 4248 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 44 packets, 4248 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all  --  *      !br-cac172d73716  172.18.0.0/16        0.0.0.0/0           
    0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0           
    0     0 MASQUERADE  tcp  --  *      *       172.18.0.2           172.18.0.2           tcp dpt:80

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  br-cac172d73716 *       0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0           
    0     0 DNAT       tcp  --  !br-cac172d73716 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8082 to:172.18.0.2:80


*filter
Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  !br-cac172d73716 br-cac172d73716  0.0.0.0/0            172.18.0.2           tcp dpt:80

with: docker network ls

NETWORK ID          NAME                DRIVER              SCOPE
cac172d73716        docker_webapp       bridge              local

Answer 1

TLDR: Due to compatibility reasons docker runs a TCP forwarding proxy by default, --userland-proxy=false switches port exposure entirely to iptables

---

Docker also starts a proxy-server by default. This service is a simple TCP forwarding proxy, accepting connections on the ^*filterINPUT-chain and forwards it into the container.

According to this article, these are the main reasons for this proxy to exist:

• To ^*filterFORWARD rules, the device must be configured for routing, eg. net.ipv4.ip_forward must be set to 1. This is not enabled by default.

• Linux doesn't allow routing on the loopback interface (lo) by default, this can be enabled by setting net.ipv4.route_localnet to 1

• The people at docker wanted to support RHEL 6.5 which runs a 2.6 kernel. As discussed here, there exist cases where a packet has to be returned to the bridge it came from. This is only possible with bridge hairpinning which was not available on this system.

ss -lnp | grep 8082 shows:

Netid  State   Recv-Q  Send-Q  Local Address:Port  Peer Address:Port
tcp    LISTEN  0       128     *:8082              *:*                users:(("docker-proxy",pid=3066,fd=4))

---

If the kernel supports bridge hairpinning the switch --userland-proxy=false can be used. The docker manual states:

The --userland-proxy parameter, true by default, provides a userland implementation for inter-container and outside-to-container communication. When disabled, Docker uses both an additional MASQUERADE iptable rule and the net.ipv4.route_localnet kernel parameter which allow the host machine to connect to a local container exposed port through the commonly used loopback address: this alternative is preferred for performance reasons.