Correct iptables rules to allow linking of docker containers

Question

I have two Docker containers on a Debian 8.3 node. One is the official postgres image, one is a basic phoenix / elixir app. I'm linking both with a docker stack file. But phoenix is not able to connect to postgres, unless I publish the port. This lets me think that something with the internal docker network is wrong and since the node is a fresh Debian install it might be the iptables. It also kind of excludes that the password or hostname is wrong.

What am I doing wrong and what is the correct way to set up the iptables rules to allow two containers to communicate?

Error message in phoenix app:

app-1 | 2016-03-15T16:15:18.019402549Z ** (Mix) The database for App.Repo couldn't be created, reason given: psql: could not connect to server: Connection refused
app-1 | 2016-03-15T16:15:18.019456447Z  Is the server running on host "postgres" (10.7.0.1) and accepting
app-1 | 2016-03-15T16:15:18.019468609Z  TCP/IP connections on port 5432?

Output of logfile in postgres container

postgres-1 | 2016-03-15T16:46:32.457844697Z LOG:  MultiXact member wraparound protections are now enabled
postgres-1 | 2016-03-15T16:46:32.464806051Z LOG:  database system is ready to accept connections
postgres-1 | 2016-03-15T16:46:32.465087076Z LOG:  autovacuum launcher started

My Docker Stack File

app:
  image: myrepo/app
  environment:
    POSTGRES_USER: postgres
    POSTGRES_PASSWORD: postgrespassword
    PORT: 4000
  links:
    - postgres
  ports:
    - 80:4000
postgres:
  image: postgres:9.5
  environment:
    POSTGRES_USER: postgres
    POSTGRES_PASSWORD: postgrespassword
  volumes:
    - /var/dbdata:/var/lib/postgresql/data

Database config in Phoenix App (prod.secret.exs)

config :data_bucket, DataBucket.Repo,
  adapter: Ecto.Adapters.Postgres,
  username: System.get_env("POSTGRES_USER"),
  password: System.get_env("POSTGRES_PASSWORD"),
  database: "app_prod",
  hostname: "postgres",
  pool_size: 20

Result of $ sudo iptables -L

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere
REJECT     all  --  anywhere             loopback/8           reject-with icmp-port-unreachable
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:2375
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:6783
ACCEPT     udp  --  anywhere             anywhere             udp dpt:6783
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:ssh
ACCEPT     icmp --  anywhere             anywhere             icmp echo-request
LOG        all  --  anywhere             anywhere             limit: avg 5/min burst 5 LOG level debug prefix "iptables denied: "
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
DROP       tcp  --  anywhere             172.17.0.1           tcp dpt:6783
DROP       udp  --  anywhere             172.17.0.1           udp dpt:6783
DROP       udp  --  anywhere             172.17.0.1           udp dpt:6784
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere

Chain DOCKER (1 references)
target     prot opt source               destination

Result of $ sudo docker version

Client:
 Version:      1.9.1-cs2
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   4ade326
 Built:        Mon Nov 30 21:56:07 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1-cs2
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   4ade326
 Built:        Mon Nov 30 21:56:07 UTC 2015
 OS/Arch:      linux/amd64

Result of sudo docker ps:

CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS              PORTS                  NAMES
f113435b781b        myrepo/app:latest       "elixir --erl '-smp d"   35 seconds ago      Up 34 seconds       0.0.0.0:80->4000/tcp   app-1.App.15ffa2c2
6e2879fd9f2c        postgres:9.5            "/docker-entrypoint.s"   37 minutes ago      Up 5 minutes        5432/tcp               postgres-1.App.cbe400ac

For the last output: The Phoenix App of course only starts if I don't run migrations. Normally this is at the end of my Dockerfile:

CMD ["elixir", "--erl", "-smp disable", "/usr/local/bin/mix", "do", "compile", ",", "ecto.create", ",", "ecto.migrate", ",", "phoenix.server"]

Which I changed to the following to get this output

CMD ["elixir", "--erl", "-smp disable", "/usr/local/bin/mix", "do", "compile", ",", "phoenix.server"]

Answer 1

I found the solution: The problem were the iptables settings. I had a rule in there to drop all forwarding unless otherwise defined:

-A FORWARD -j REJECT

Getting rid of this rule was the solution.

Edit:

As @Zoredache pointed out just deleting the rule is obviously a bad idea and I'm an idiot. The correct way is to put it at the very end of the rule chain, after the rules created by docker. So if you have the same problem, this works for me:

Get rid of the rule

$ sudo iptables -D FORWARD -j REJECT

Add it again to move it to the end of the set

$ sudo iptables -A FORWARD -j REJECT

Make sure they are in the right order. So the reject all else rule should be at the very end.

$ sudo iptables -v -L FORWARD

On Debian this is how you can make sure the rules still apply when restarting the server:

Once you are happy, save the new rules to the master iptables file:

$ iptables-save > /etc/iptables.up.rules

To make sure the iptables rules are started on a reboot we'll create a new file:

$ editor /etc/network/if-pre-up.d/iptables

Add these lines to it:

#!/bin/sh
/sbin/iptables-restore < /etc/iptables.up.rules

The file needs to be executable so change the permissions:

$ sudo chmod +x /etc/network/if-pre-up.d/iptables