1

I have an AWS VPC with some public subnets and a private subnet, like the image below.

  • Both instances can connect to the internet (INSTANCE A connects through NAT GATEWAY instance)
  • NAT GATEWAY can ping and traceroute hosts on internet and instances on other subnets
  • INSTANCE A can ping NAT GATEWAY and other instances in its subnet and other subnets

The NAT GATEWAY is a Ubuntu 16.04 (t2.micro) instance, configured by me. It's not a managed AWS NAT gateway. It's working perfectly as a gateway for all other hosts inside the VPC, as well for D-NAT (for some private Apache servers) and also acting as a SSH Bastion.

The problem is that INSTANCE A can't ping or traceroute hosts on internet. I've already tried/checked:

  • Route tables
  • Security groups
  • IPTABLES rules
  • kernel parameters

vpc diagram

Security Groups

 NAT GATEWAY
 Outbound: 
  * all traffic allowed
 Inbound:
  * SSH from 192.168.0.0/16 (VPN network)
  * HTTP/S from 172.20.0.0/16 (allowing instances to connect to the internet)
  * HTTP/S from 0.0.0.0/0 (allowing clients to access internal Apache servers through D-NAT)
  * ALL ICMP V4 from 0.0.0.0/0 


 INSTANCE A
 Outbound: 
  * all traffic allowed
 Inbound:
  * SSH from NAT GATEWAY SG
  * HTTP/S from 172.20.0.0/16 (public internet throught D-NAT)
  * ALL ICMP V4 from 0.0.0.0/0

Route tables

PUBLIC SUBNET
172.20.0.0/16: local
0.0.0.0/0: igw-xxxxx (AWS internet gateway attached to VPC)

PRIVATE SUBNET
0.0.0.0/0: eni-xxxxx (network interface of the NAT gateway)
172.20.0.0/16: local

Iptables rules

# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

# iptables -tnat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A POSTROUTING -o eth0 -j MASQUERADE

Kernel parameters

net.ipv4.conf.all.accept_redirects = 0  # tried 1 too
net.ipv4.conf.all.secure_redirects = 1
net.ipv4.conf.all.send_redirects = 0  # tried 1 too
net.ipv4.conf.eth0.accept_redirects = 0  # tried 1 too
net.ipv4.conf.eth0.secure_redirects = 1
net.ipv4.conf.eth0.send_redirects = 0  # tried 1 too
net.ipv4.ip_forward = 1

Sample traceroute from INSTANCE A

Thanks to @hargut for pointing out a detail about traceroute using UDP (and my SGs not allowing it). So, using it with -I option for ICMP:

# traceroute -I 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  ip-172-20-16-10.ec2.internal (172.20.16.10)  0.670 ms  0.677 ms  0.700 ms
 2  * * *
 3  * * *
 ...
3
  • You said your "NAT Gateway can ping and traceroute hosts on internet" - how did you manage to get a shell on a managed service to do that? Suggest you review your question to make sure it's precise and accurate. Perhaps you could also show screenshots of your security groups, NACLs, and route tables to validate the information you've given us. I would try opening ICMP more widely to start. If an instance using the internet gateway directly can ping the internet and an instance in the private subnet can't then it's likely a problem with NAT, SG, NACL, or route tables as you said.
    – Tim
    Sep 5, 2018 at 19:31
  • @Tim it's a custom NAT gateway, running on a t2.micro instance with Ubuntu 16.04 - I'll update the question with this info and the other suggested. This diagram is a simplification of the actual network, with some info redacted - I'm unable to post actual screenshots here. But you can be sure I've only taken out irrelevant rules (like SG rules allowing http traffic). I'm expecting it to be something in iptables or linux kernel. Sep 5, 2018 at 20:09
  • @DanielCosta, did you find a solution yet? I got into the same issue and I wonder what the issue is. Thanks
    – Nam Nguyen
    Oct 17, 2018 at 23:23

1 Answer 1

0

Linux/Unix traceroute uses UDP for standard requests. Your security group does not allow UDP inbound packets.

From the traceroute man page:

In the modern network environment the traditional traceroute methods can not be always applicable, because of widespread use of firewalls. Such firewalls filter the "unlikely" UDP ports, or even ICMP echoes. To solve this, some additional tracerouting methods are implemented (including tcp), see LIST OF AVAILABLE METHODS below. Such methods try to use particular protocol and source/destination port, in order to bypass firewalls (to be seen by firewalls just as a start of allowed type of a network session)

https://linux.die.net/man/8/traceroute

See the -I option of tracerout which switches the traceroute mode to ICMP based tracing.

5
  • Indeed, when using -I option, traceroute is able to list the first hop (which is the NAT gateway) but can't go beyond that. On the other hand, ping still doesn't works. Sep 5, 2018 at 21:19
  • On iptables side I don't see any reason why http (tcp) traffic should work, and why icmp would not. Are the "Source Destination Checks" on AWS EC2 level disabled? How about kernel level rp_filter settings?
    – hargut
    Sep 5, 2018 at 21:47
  • "Source destination Checks" are disabled - double checked that. rp_filter was set to 1 on eth0, default and all. Tried setting all of them to 0 but didn't work either. Sep 5, 2018 at 22:01
  • How do the full routing tables look like? ip route list table all How about static rules on subnet/vpc (AWS Network ACL) level?
    – hargut
    Sep 5, 2018 at 22:11
  • Route tables on subnet/vpc are detailed on my post above. For ip route output see (pastebin.com/CnXPXg2g) Sep 5, 2018 at 22:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.