0

On AWS, I have run a consul container and a overlay network on a Docker host.

 docker ps
CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS              PORTS                                                                            NAMES
77b97d803d8c        busybox                     "sh"                     5 days ago          Up 5 days                                                                                            busybox
206af8b467c2        daocloud.io/library/httpd   "httpd-foreground"       5 days ago          Up 5 days           0.0.0.0:32768->80/tcp                                                            web1
b40d24fd8f91        progrium/consul             "/bin/start -serve..."   5 days ago          Up 5 days           53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp   consul

docker network inspect myoverlay
[
    {
        "Name": "myoverlay",
        "Id": "a1752c608259d44a5bc0ad8b47e8b340d0252d026bf113cea681e8a92633e844",
        "Created": "2017-09-05T06:37:13.271539119Z",
        "Scope": "global",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.16.16.0/24",
                    "IPRange": "172.16.16.128/25",
                    "AuxiliaryAddresses": {
                        "ip2": "172.16.16.2"
                    }
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "206af8b467c2020b51017524984b322f1544e6b8594b672d4fa6bcb7b263b564": {
                "Name": "web1",
                "EndpointID": "d44ecac9eb248a90dd895d31170aca65333f248616a0998e22b81e4a1c7414b2",
                "MacAddress": "02:42:ac:10:10:81",
                "IPv4Address": "172.16.16.129/24",
                "IPv6Address": ""
            },
            "77b97d803d8c0c3c3ceb0bcb6ca4f82971f9a68273ec32266cadd950350a4a7e": {
                "Name": "busybox",
                "EndpointID": "1a00a326b49b448c7b280faca58bb7ca67cea5d11c513ce64928b0a76fb673f3",
                "MacAddress": "02:42:ac:10:10:82",
                "IPv4Address": "172.16.16.130/24",
                "IPv6Address": ""
            },
            "ep-f774da45dbf1655150fe5dd0a76b29c02873dd79181f26d1c39f12fd2523897d": {
                "Name": "web2",
                "EndpointID": "f774da45dbf1655150fe5dd0a76b29c02873dd79181f26d1c39f12fd2523897d",
                "MacAddress": "02:42:ac:10:10:83",
                "IPv4Address": "172.16.16.131/24",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

As you can see, there are three containers running on the Docker host that is running the Consul container. And the containers within one Docker host can ping each other with the overlay IP.

docker exec 77b97d803d8c ping 172.16.16.129
PING 172.16.16.129 (172.16.16.129): 56 data bytes
64 bytes from 172.16.16.129: seq=0 ttl=64 time=0.098 ms
64 bytes from 172.16.16.129: seq=1 ttl=64 time=0.096 ms
64 bytes from 172.16.16.129: seq=2 ttl=64 time=0.096 ms
64 bytes from 172.16.16.129: seq=3 ttl=64 time=0.098 ms
^C

The following output is from another Docker host that has joined in the overlay network.

 docker network inspect myoverlay
[
    {
        "Name": "myoverlay",
        "Id": "a1752c608259d44a5bc0ad8b47e8b340d0252d026bf113cea681e8a92633e844",
        "Created": "2017-09-05T06:37:13.271539119Z",
        "Scope": "global",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.16.16.0/24",
                    "IPRange": "172.16.16.128/25",
                    "AuxiliaryAddresses": {
                        "ip2": "172.16.16.2"
                    }
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "9caaf8d7fe01104f4609053d94ca231fc4c91d5862b90244768c2c08f99cb65e": {
                "Name": "web2",
                "EndpointID": "f774da45dbf1655150fe5dd0a76b29c02873dd79181f26d1c39f12fd2523897d",
                "MacAddress": "02:42:ac:10:10:83",
                "IPv4Address": "172.16.16.131/24",
                "IPv6Address": ""
            },
            "ep-1a00a326b49b448c7b280faca58bb7ca67cea5d11c513ce64928b0a76fb673f3": {
                "Name": "busybox",
                "EndpointID": "1a00a326b49b448c7b280faca58bb7ca67cea5d11c513ce64928b0a76fb673f3",
                "MacAddress": "02:42:ac:10:10:82",
                "IPv4Address": "172.16.16.130/24",
                "IPv6Address": ""
            },
            "ep-d44ecac9eb248a90dd895d31170aca65333f248616a0998e22b81e4a1c7414b2": {
                "Name": "web1",
                "EndpointID": "d44ecac9eb248a90dd895d31170aca65333f248616a0998e22b81e4a1c7414b2",
                "MacAddress": "02:42:ac:10:10:81",
                "IPv4Address": "172.16.16.129/24",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

The docker containers from one Docker host somehow can not ping the ones in another Docker host.

docker exec 77b97d803d8c ping 172.16.16.131

The security group of the Docker hosts have allowed all traffic from each other.

I tried to telnet 7946 port of Serf of one Docker host, but it only returns no route to host.

Is there anything wrong with my configuration? Or any lack of configuration for the overlay? Thanks in advance for any help.

2

The behavior you're seeing suggests that there is a firewall somewhere between the docker hosts, maybe a firewall on the host itself, or perhaps on the network. You need the following open:

  • 7946/both (control)
  • 4789/udp (data)
  • protocol 50 for ipsec

The iptables commands for this (if you have iptables defaulting to drop):

iptables -A INPUT -p tcp -m tcp --dport 7946 -j ACCEPT
iptables -A INPUT -p udp -m udp --dport 7946 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 4789 -j ACCEPT
iptables -A INPUT -p 50 -j ACCEPT

I'd recommend using tcpdump on each of the nodes to monitor for whether this traffic is being blocked somewhere.

The other possibility is that you're encountering a defect with Docker, like issue 32195. However, that would only affect some requests occasionally, not every request.

  • Thanks. It indeed is caused by iptables misconfiguration. – Jepsenwan Oct 1 '17 at 1:12
  • @BMitch, should not second command be iptables -A INPUT -p udp -m udp --dport 7946 -j ACCEPT and third iptables -A INPUT -p udp -m udp --dport 4789 -j ACCEPT? – heroin Dec 19 '17 at 8:59
  • @heroin indeed it should. Good catch. – BMitch Dec 19 '17 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.