2

I have created a four subnet VPC, where two subnets are private and two public. Instances in the private subnet has only private IP's and reach the the internet through an IGW/NAT GW.

I can successfully create an ECS clusters spanning my public subnets, but when I create a cluster in the backend subnets, the cluster instances are unable to register in the cluster.

I suspect network issues, and I am debugging with Flowlogs. In the mean time, I was hoping someone could shed some light over what ports that need to be open, or other configuration involved in ECS cluster management.

2
  • I don't know much about ECS. Try opening all ports / routes to your public subnet, if it makes a difference that could be interesting and useful diagnosis information.
    – Tim
    Feb 13, 2017 at 8:01
  • My analysis so far shows a lot of blocked UDP traffic from cluster instances to hosts that are not a part of my infra.
    – Glenn Bech
    Feb 13, 2017 at 9:13

2 Answers 2

5

I suspect this is port related, as it's fairly standard that things in a cluster need to communicate. The two articles below should answer this question for you. It seems to me the following are probably required

  • Amazon ECS agent ports 51678 and 51679 (protocol unspecified)
  • TCP 2376 and 2377 (docker)
  • TCP / UDP 7946 (docker)
  • UDP 4789 (docker)
  • Ephemeral ports 49153 to 65535 (protocol unspecified)

Note that I know little about ECS and have simply done a couple of Google searches and read documentation. ECS is based on Docker so I looked at that. Some experimentation will be required.

Digital Ocean has a good article on Docker Ports.

TCP port 2376 for secure Docker client communication. This port is required for Docker Machine to work. Docker Machine is used to orchestrate Docker hosts. TCP port 2377. This port is used for communication between the nodes of a Docker Swarm or cluster. It only needs to be opened on manager nodes. TCP and UDP port 7946 for communication among nodes (container network discovery). UDP port 4789 for overlay network traffic (container ingress networking).

Then the Amazon documentation mentions some of the same ports.

The default ephemeral port range is 49153 to 65535, and this range is used for Docker versions prior to 1.6.0. For Docker version 1.6.0 and later, the Docker daemon tries to read the ephemeral port range from /proc/sys/net/ipv4/ip_local_port_range; if this kernel parameter is unavailable, the default ephemeral port range is used. You should not attempt to specify a host port in the ephemeral port range, because these are reserved for automatic assignment. In general, ports below 32768 are outside of the ephemeral port range.

The default reserved ports are 22 for SSH, the Docker ports 2375 and 2376, and the Amazon ECS container agent ports 51678 and 51679. Any host port that was previously specified in a running task is also reserved while the task is running (after a task stops, the host port is released).The current reserved ports are displayed in the remainingResources of DescribeContainerInstances output, and a container instance may have up to 100 reserved ports at a time, including the default reserved ports (automatically assigned ports do not count toward the 100 reserved ports limit).

Note that as per Wikipedia and comments below the ephemeral port range may need to be expanded.

4
  • 49153 to 65535 is not enough. Had to open from 32768 to 61000 according to this page (and it worked!) docs.docker.com/v17.09/engine/userguide/networking/…
    – Sanctus
    Sep 13, 2018 at 18:15
  • The range I specified is the IANA range. Linux doesn't always use the standard. Wikipedia has more info.
    – Tim
    Sep 13, 2018 at 22:46
  • I suppose question was about AWS, not about spherical ports in vacuum, so may be it makes sense to point out real situation.
    – Sanctus
    Sep 14, 2018 at 10:58
  • You write in list: TCP 2376 and 2377 (docker) , and then in description: 1) TCP port 2376, TCP port 2377. 2) The default reserved ports are 22 for SSH, the Docker ports 2375 and 2376
    – Sanctus
    Nov 13, 2018 at 15:30
1

It's important to remember that the Network ACL for the subnet that contains the NAT gateway needs a rule for inbound SSL (443). If not, no outbound SSL can be done from the private subnet. Network traffic from my private subnet to http://cloudformation.eu-central-1.amazonaws.com/ was blocked on the subnet level.

This made the shell command, running on the back-end instances, cfn-signal to block. The cfn-signal command's duty is to report back to Cloud formation that an instance is up and running, so that the script may continue to create dependent resources.

I thought of this as an ECS problem, not a Cloudformation problem.

2
  • Thanks for this answer to your question. It isn't clear to me why the Network ACL needs to permit ingress to port 443 on the NAT Gateway. Is this because some AWS components use the NAT Gateway as if it were an HTTP proxy?
    – jonseymour
    Mar 19, 2019 at 5:22
  • By default, Network ACLs allow all traffic. They're not active so to speak. If you start using them, then you'll need an ingress rule to the NAT GW or traffic will be rejected at the subnet level - in the subnet that hosts the Nat GW.
    – Glenn Bech
    Mar 25, 2019 at 9:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .