port forwarding through AWS VPC NAT

Question

Yes, I have already scoured the internet and read most of the popular IPTables / DNAT guides / pages / posts.

My Problem

Summary I have a VPC with several subnets. One subnet in particular requires an EIP for internet connectivity. I have a web server living in this subnet. I need to access it from behind a nat (also in the same subnet, has an EIP). I am attempting to set up my NAT so that incoming packets destined for a certain port are sent off to a different host (essentially i need DNAT)

Part 1: IN THE VPC:

I have SSH'd into my NAT instance through my WAN accessible SSH Proxy. Here's the default IPTables that the amazon NAT instance comes with. Nothing special, just a POSTROUTE rule as expected. The NAT instance has only one interface; eth0 (and lo, but let's pretend that does not exist)

[root@IP_NAT ec2-user]# iptables -t nat -L --line-numbers
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination     

Ok. Now let's add two port forwards, so my web server can be accessed behind the NAT

[root@IP_NAT ec2-user]# iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT --to-destination IP_WEBSERVER:80
[root@IP_NAT ec2-user]# iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j DNAT --to-destination IP_WEBSERVER:443

Right! Time to check my work:

[root@IP_NAT ec2-user]# iptables -t nat -L --line-numbers
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    DNAT       tcp  --  anywhere             anywhere             tcp dpt:http to:IP_WEBSERVER:80
2    DNAT       tcp  --  anywhere             anywhere             tcp dpt:https to:IP_WEBSERVER:443

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    MASQUERADE  all  --  ip-10-0-0-0.us-west-1.compute.internal/16  anywhere  

So, things look like they should work just fine. Except, they dont. :(. I can wget / telnet / ping / ssh my to my heart's content between the nat and the web server. I can telnet on some ports to my NAT, but others just time out. I have triple checked that the AWS Security Group governing the NAT instance allows the correct ports in and out. Just for testing, I allowed all traffic on all ports both in / out. I still have the same problem.

The webserver shows no activity on any of it's logs.

Here's a little bit of output from my local machine showing what has been bugging me all day:

LOCAL_USER@LOCAL_HOST:~$ telnet AWS_EIP 443
Trying AWS_EIP...
^C                (time out, here)
LOCAL_USER@LOCAL_HOST:~$ telnet AWS_EIP 80
Trying AWS_EIP...
^C                (time out, here)
LOCAL_USER@LOCAL_HOST:~$ telnet AWS_EIP 22
Trying AWS_EIP...
Connected to AWS_EIP.
Escape character is '^]'.
SSH-2.0-OpenSSH_6.2
asd
Protocol mismatch.
Connection closed by foreign host.

So when I try to open a TCP connection on port 22, things work instantly. Why is my nat not 'listening' on port 443 or 80, despite iptables explicitly allowing traffic on these ports?

As a last bit of useful information, here's the sysctl.conf file:

[root@IP_NAT ec2-user]# cat /etc/sysctl.conf 

# Controls IP packet forwarding
net.ipv4.ip_forward = 1

# Controls source route verification
net.ipv4.conf.default.rp_filter = 0

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 1

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

Any thoughts as to why packets don't even make it to my webserver?

Answer 1

From the information in your question, I assume you have 2 EIP's in use. One for the NAT server and one for the Web server. If that is the case, then the NAT server and everyone in the world should be able to connect to the Web server via its EIP fine, assuming all firewall/security groups are correct.

Now, I'm confused what the NAT server is actually for. I assume that you want a group of hosts behind this NAT to be able to access the Web server AND you don't want all those hosts to each have its own EIP? If that's the case, you'll need two VPC subnets and the NAT server needs to do SNAT (which seems to be what you are doing with the MASQUERADE rule). For this setup, it's documented in http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_NAT_Instance.html

Basic requirements are (and I've done this before at a previous job):

• A NAT Gateway server is required. IP forwarding must be enabled along with a MASQUERADE rule in iptables (like what you have above, without the DNAT rules).
• At least two subnets are required:
  • One "public" subnet where that subnet's default route in the VPC route table points to the Internet Gateway. All hosts in this subnet requires an EIP to access the Internet. The NAT Gateway server must be in this subnet.
  • One "private" subnet where its default route points to the NAT Gateway server. That's the network hidden by the NAT, much like a DSL router at a home network. All hosts in that subnet will access the Internet (or hosts in the public subnet if public EIP addresses are used instead of their private RFC1918 addresses) via the NAT and its EIP.

Lastly, if you want to avoid giving the web server an EIP, then one thing I would try (though I haven't tested it personally) is to move the web server into the private subnet and change your DNAT rules to point to its internal address. Obviously, the VPC subnets, routes, etc. needs to be configured per the AWS doc above.

Answer 2

[root@IP_NAT ec2-user]# iptables -t nat -L --line-numbers
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    DNAT       tcp  --  anywhere             anywhere             tcp dpt:http to:IP_WEBSERVER:80
2    DNAT       tcp  --  anywhere             anywhere             tcp dpt:https to:IP_WEBSERVER:443

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    MASQUERADE  all  --  ip-10-0-0-0.us-west-1.compute.internal/16  anywhere  

ANSWER

For Future people encountering this issue: ensure your security group allows you to come back in over the correct public IP. I entered the CIDR range of the incoming IPs and this worked for me using all the same configuration above. Note: the Public IP of your web app or nat instance will not be the same as the ip coming back in I found it via a nestat.