0

I have a server that I want to be accessible on the public Internet via SSH only, but I want it to be able to connect out to a VPN, and bind services to its VPN interface.

I'm using Linode to host a Ubuntu 16.04.2 LTS VM, ZeroTier for my VPN layer, and Docker in Swarm mode (with just a single node) for the services.

Running ifconfig -a on the VM returns this (public IPs redacted for privacy):

docker0   Link encap:Ethernet  HWaddr 02:42:f7:e7:e6:1e
          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

docker_gwbridge Link encap:Ethernet  HWaddr 02:42:ad:82:56:0f
          inet addr:172.18.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:adff:fe82:560f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6083 errors:0 dropped:0 overruns:0 frame:0
          TX packets:16927 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:6364682 (6.3 MB)  TX bytes:49725061 (49.7 MB)

eth0      Link encap:Ethernet  HWaddr f2:3c:91:a7:c3:ac
          inet addr:<REDACTED>  Bcast:<REDACTED>  Mask:255.255.255.0
          inet6 addr: <REDACTED>/64 Scope:Global
          inet6 addr: <REDACTED>/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:94675 errors:0 dropped:0 overruns:0 frame:0
          TX packets:47778 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:59390529 (59.3 MB)  TX bytes:14765979 (14.7 MB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:54171 errors:0 dropped:0 overruns:0 frame:0
          TX packets:54171 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:6243573 (6.2 MB)  TX bytes:6243573 (6.2 MB)

veth8323309 Link encap:Ethernet  HWaddr 82:b3:ec:d8:8c:9b
          inet6 addr: fe80::80b3:ecff:fed8:8c9b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:15 errors:0 dropped:0 overruns:0 frame:0
          TX packets:538 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1248 (1.2 KB)  TX bytes:22932 (22.9 KB)

veth602daf7 Link encap:Ethernet  HWaddr 82:7e:cf:b3:cb:a4
          inet6 addr: fe80::807e:cfff:feb3:cba4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3065 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11154 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:6187902 (6.1 MB)  TX bytes:794193 (794.1 KB)

zt0       Link encap:Ethernet  HWaddr 42:08:c2:e8:fc:93
          inet addr:192.168.196.106  Bcast:192.168.196.255  Mask:255.255.255.0
          inet6 addr: fe80::4008:c2ff:fee8:fc93/64 Scope:Link
          inet6 addr: fc7b:7a95:194:6f84:bf9a::1/40 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:2800  Metric:1
          RX packets:442 errors:0 dropped:0 overruns:0 frame:0
          TX packets:279 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:52165 (52.1 KB)  TX bytes:46603 (46.6 KB)

My Docker compose file looks like this:

version: "3"
services:
  jenkins:
    image: jenkins/jenkins:lts
    ports:
      - "80:8080"
    volumes:
      - jenkins_home:/var/jenkins_home
    deploy:
      replicas: 1
volumes:
  jenkins_home:

However, I can't seem to get the iptables rules right. I had expected to be able to do this:

# Allow established connections, including outbound connections this server initiated, to continue to receive replies
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Allow SSH on eth0
iptables -A INPUT -i eth0 -p tcp --dport 22 -j ACCEPT

# Allow UDP 9993 on eth0 (required for ZeroTier)
iptables -A INPUT -i eth0 -p udp --dport 9993 -j ACCEPT
iptables -A OUTPUT -p udp --dport 9993 -j ACCEPT

# Reject everything that hasn't already been accepted (rules are applied in order)
iptables -A INPUT -i eth0 -j REJECT

However, when I deploy and start my Docker Swarm service, I end up being able to access the service via the address on eth0 from a remote host that's not on the VPN, from the VPN address on zt0 from a remote host that is on the VPN, but not from localhost on the VM itself.

What am I doing wrong here? How can I get iptables rules that will prevent traffic from the public Internet but allow it from only the ZeroTier VPN?

0

After more research, I found my problem.

The gist of it is that Docker uses its own chains for forwarding packets. It will modify the DOCKER and DOCKER-INGRESS chains when the Docker daemon starts, and when it deploys new containers, but it will not modify a chain called DOCKER-USER that is intended for administrator-defined firewall rules of the type I want to apply.

Armed with that knowledge, I was able to get what I wanted with the following iptables rules:

#!/bin/sh

set -e
set -u

# REMINDER: DROP vs REJECT:
# a DROP operation blackholes the packet, likely causing client to wait dozens of seconds for a timeout
# a REJECT operation sends an ICMP response letting the client know the packet was rejected

# REMINDER: IPv4 vs IPv6
# `iptables` only applies IPv4 rules; `ip6tables` only lapplies IPv6 rules.

## System firewall rules

# Allow any established sessions from our container host - including replies to outbound queries - to receive traffic
iptables --append INPUT -m conntrack --ctstate ESTABLISHED,RELATED --jump ACCEPT
ip6tables --append INPUT -m conntrack --ctstate ESTABLISHED,RELATED --jump ACCEPT

# Allow SSH on eth0
iptables --append INPUT --in-interface eth0 --protocol tcp --dport 22 --jump ACCEPT
ip6tables --append INPUT --in-interface eth0 --protocol tcp --dport 22 --jump ACCEPT

# Allow UDP 9993 on eth0 (required for ZeroTier)
iptables --append INPUT --in-interface eth0 --protocol udp --dport 9993 --jump ACCEPT
ip6tables --append INPUT --in-interface eth0 --protocol udp --dport 9993 --jump ACCEPT
iptables --append OUTPUT --protocol udp --dport 9993 --jump ACCEPT
ip6tables --append OUTPUT --protocol udp --dport 9993 --jump ACCEPT

# Reject everything to eth0 that hasn't been accepted by a previous rule
iptables --append INPUT --in-interface eth0 --jump REJECT
ip6tables --append INPUT --in-interface eth0 --jump REJECT

## Docker firewall rules

# NOTE: Docker does not enable IPv6 networking by default, so there is no need for `ip6tables` here
# <https://docs.docker.com/engine/userguide/networking/default_network/ipv6/>

# NOTE: Docker promises not to modify the DOCKER-USER chain so that we can do this type of filtering

# Create the chain we need if Docker hasn't done so already
iptables --new DOCKER-USER 2>/dev/null || true

# Allow any established sessions from our containers - including replies to outbound queries - to receive traffic
iptables --append DOCKER-USER --match conntrack --ctstate ESTABLISHED,RELATED --jump ACCEPT

# Reject packets that would be forwarded to Docker when they come in over eth0 (the public interface)
iptables --append DOCKER-USER --in-interface eth0 ---jump REJECT

Note that some of these rules are not specific to Docker, but exist to reject packets intended for my Docker server - I only want to accept VPN and SSH packets to the server itself.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.