No network connectivity to/from Docker CE container on CentOS 8

Question

I just installed the latest release of docker-ce on CentOS, but I can't reach published ports from a neighboring server and can't reach the outside from the container itself.

Running a plain vanilla CentOS 8 with NetworkManager and FirewallD enabled. Default firewall zone is public.

Versions:

• docker-ce 19.03.3 (official Docker RPM)
• containerd.io 1.2.6 (official Docker RPM for CentOS 7 - not available for CentOS 8 yet)
• CentOS 8.0.1905 (minimal install)

Answer 1

After spending a couple of days looking at logs and configurations for the involved components, I was about to throw in the towel and revert back to Fedora 30, where this seems to work straight out of the box.

Focusing on firewalling, I realized that disabling firewalld seemed to do the trick, but I would prefer not to do that. While inspecting network rules with iptables, I realized that the switch to nftables means that iptables is now an abstraction layer that only shows a small part of the nftables rules. That means most - if not all - of the firewalld configuration will be applied outside the scope of iptables.

I was used to be able to find the whole truth in iptables, so this will take some getting used to.

Long story short - for this to work, I had to enable masquerading. It looked like dockerd already did this through iptables, but apparently this needs to be specifically enabled for the firewall zone for iptables masquerading to work:

# Masquerading allows for docker ingress and egress (this is the juicy bit)
firewall-cmd --zone=public --add-masquerade --permanent

# Specifically allow incoming traffic on port 80/443 (nothing new here)
firewall-cmd --zone=public --add-port=80/tcp
firewall-cmd --zone=public --add-port=443/tcp

# Reload firewall to apply permanent rules
firewall-cmd --reload

Reboot or restart dockerd, and both ingress and egress should work.

Answer 2

What's missing from the answers before is the fact that you first need to add your docker interface to the zone you configure, e.g. public (or add it to the "trusted" zone which was suggested here but I doubt that's wise, from a security perspective). Because by default it's not assigned to a zone. Also remember to reload the docker daemon when done.

# Check what interface docker is using, e.g. 'docker0'
ip link show

# Check available firewalld zones, e.g. 'public'
sudo firewall-cmd --get-active-zones

# Check what zone the docker interface it bound to, most likely 'no zone' yet
sudo firewall-cmd --get-zone-of-interface=docker0

# So add the 'docker0' interface to the 'public' zone. Changes will be visible only after firewalld reload
sudo nmcli connection modify docker0 connection.zone public

# Masquerading allows for docker ingress and egress (this is the juicy bit)
sudo firewall-cmd --zone=public --add-masquerade --permanent
# Optional open required incomming ports (wasn't required in my tests)
# sudo firewall-cmd --zone=public --add-port=443/tcp
# Reload firewalld
sudo firewall-cmd --reload
# Reload dockerd
sudo systemctl restart docker

# Test ping and DNS works:
docker run busybox ping -c 1 172.16.0.1
docker run busybox cat /etc/resolv.conf
docker run busybox ping -c 1 yourhost.local

Answer 3

To be able to set fine-grained rules for Docker, I did not need to set docker0 to any zone.

# 1. Stop Docker
systemctl stop docker

# 2. Recreate DOCKER-USER chain in firewalld. 
firewall-cmd --permanent \
             --direct \
             --remove-chain ipv4 filter DOCKER-USER

firewall-cmd --permanent \
             --direct \
             --remove-rules ipv4 filter DOCKER-USER

firewall-cmd --permanent \
             --direct \
             --add-chain ipv4 filter DOCKER-USER

# (Ignore any warnings)

# 3. Docker Container <-> Container communication

firewall-cmd --permanent \
             --direct \
             --add-rule ipv4 filter DOCKER-USER 1 \
             -m conntrack --ctstate RELATED,ESTABLISHED \
             -j ACCEPT \
             -m comment \
             --comment 'Allow docker containers to connect to the outside world'

firewall-cmd --permanent \
             --direct \
             --add-rule ipv4 filter DOCKER-USER 1 \
             -j RETURN \
             -s 172.17.0.0/16 \
             -m comment \
             --comment 'allow internal docker communication'

# Change the Docker Subnet to your actual one (e.g. 172.18.0.0/16)

# 4. Add rules for IPs allowed to access the Docker exposed ports.

firewall-cmd --permanent \
             --direct \
             --add-rule ipv4 filter DOCKER-USER 1 \
             -o docker0 \
             -p tcp \
             -m multiport \
             --dports 80,443 \
             -i eth0 \
             -o docker0 \
             -s 1.2.3.4/32 \
             -j ACCEPT \
             -m comment \
             --comment 'Allow IP 1.2.3.4 to docker ports 80 and 443'

# 5. log docker traffic (if you like)

firewall-cmd --direct \
             --add-rule ipv4 filter DOCKER-USER 0 \
             -j LOG \
             --log-prefix ' DOCKER: '

# 6. Block all other IPs. 
This rule has lowest precedence, so you can add allowed IP rules later.

firewall-cmd --permanent \
             --direct \
             --add-rule ipv4 filter DOCKER-USER 10 \
             -j REJECT \
             -m comment \
             --comment 'reject all other traffic to DOCKER-USER'

# 7. Reload firewalld, Start Docker again
firewall-cmd --reload
systemctl start docker

This ends in rules defined in /etc/firewalld/direct.xml:

<?xml version="1.0" encoding="utf-8"?>
<direct>
  <chain ipv="ipv4" table="filter" chain="DOCKER-USER"/>
  <rule ipv="ipv4" table="filter" chain="DOCKER-USER" priority="0">-m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -m comment --comment 'Allow docker containers to connect to the outside world'</rule>
  <rule ipv="ipv4" table="filter" chain="DOCKER-USER" priority="0">-j RETURN -s 172.17.0.0/16 -m comment --comment 'allow internal docker communication'</rule>
  <rule ipv="ipv4" table="filter" chain="DOCKER-USER" priority="0">-p tcp -m multiport --dports 80,443 -s 1.2.3.4/32 -j ACCEPT -m comment --comment 'Allow IP 1.2.3.4 to docker ports 80 and 443'</rule>
  <rule ipv="ipv4" table="filter" chain="DOCKER-USER" priority="0">-j LOG --log-prefix ' DOCKER TCP: '</rule>
  <rule ipv="ipv4" table="filter" chain="DOCKER-USER" priority="10">-j REJECT -m comment --comment 'reject all other traffic to DOCKER-USER'</rule>
</direct>

Drawback still is that you need to install containerd.io from CentOS7 as stated by Saustrup

Answer 4

I have changed the FirewallBackend variable to iptables again and it works for me.

With this update, the nftables filtering subsystem is the default firewall backend for the firewalld daemon. To change the backend, use the FirewallBackend option in the /etc/firewalld.conf file.

Link: Centos8 Deprecated_functionality

I don't have too much information about this behavior change. Some of the iptables rules that Docker tries to use are not working according to the CentOS8 logs:

WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).