From within a docker container (running ubuntu 18) running on AWS ECS I'm attempting to establish a connection to an outside data center. We've troubleshot the issue to where we believe it's the extra hop added by the local docker network that is causing the failure. This is supported by the fact that a curl request to the destination IP completes successfully from the docker host EC2 instance, as well as from inside the same docker container when deployed to a subnet that is less than 33 hops from the destination IP.
traceroute <destination_ip> from within the container I see 33 hops:
root@1cfbdf43c8f5:~# traceroute -m36 <destination_ip> traceroute to <destination_ip> (<destination_ip>), 36 hops max, 60 byte packets 1 ip-172-17-0-1.us-east-2.compute.internal (172.17.0.1) 0.039 ms 0.014 ms 0.013 ms 2 ip-10-133-216-197.us-east-2.compute.internal (10.133.216.197) 1.185 ms 1.146 ms 1.107 ms 3 ec2-52-15-0-157.us-east-2.compute.amazonaws.com (220.127.116.11) 8.188 ms ec2-52-15-0-169.us-east-2.compute.amazonaws.com (18.104.22.168) 5.615 ms ec2-52-15-0-161.us-east-2.compute.amazonaws.com (22.214.171.124) 10.227 ms ... 32 <destination_ip> 24.706 ms 24.584 ms 24.698 ms 33 <destination_ip> 24.411 ms 24.426 ms 24.323 ms
The first hop is docker, second is the AWS NAT gateway before winding its way through the AWS networks and finally arriving at at hop 33.
curl <destination_address> while capturing with
tcpdump -v host <destination_ip> on the EC2 host machine running docker, I see that the request fails due to ttl:
ip-10-133-218-86.us-east-2.compute.internal > <destination_ip>: ICMP time exceeded in-transit, length 52
However, the inspection of that same
tcpdump shows the request has a TTL of 63 as it passes through the host, indicating it is correctly using the ubuntu system default of 64:
Time to live: 63
My question is: what may cause a request being sent with a TTL of 64 to fail on connection to a destination IP that traceroute shows is only 33 away?
It seems our options at this point are to (1) decrease the number of hops between source and destination, or else (2) increase the TTL of the outgoing request.
In an attempt to do (2), increase the TTL, I've tried updating the sys property
/proc/sys/net/ipv4/ip_default_ttl=128. tcpdump inspection shows this is being respected in the outgoing request, however the call still fails with
ICMP time exceeded in-transit.
As the answer points out, the [SYN,ACK] response has a TTL that is too low to reach back to the machine initiating the request. In the image of me hitting that same server locally, you can see it is about 200 hops fewer than any other response by that server.