My AWS development has the following architecture.
1 VPC having 2 Private Subnets: 1 for RDS (10.0.11.0/24), 1 for EC2(10.0.1.0/24) 1 Public Subnet:
1 Internet Gateway attached to my VPC
3 Route Table:
a) Main Route Table (0.0.0.0/0 target Internet Gateway)
b) EC2 Route Table (Associated with EC2 Private Subnet) plus additional route 0.0.0.0/0 targeted to Network ID interface of NAT instance
c) RDS Route Table (Associated with RDS Private Subnet)
3 Security Groups:
a) EC2 (Incoming as selected and Outgoing allowed All)
b) RDS (Incoming as selected and Outgoing allowed All)
c) NAT (Incoming as selected and Outgoing allowed All)
iptable prerouting rules have been added in NAT instance to redirect data to EC2 instance ports. Incoming data is fine as per the rules. SSH and other opened ports are working.
Problem is all outgoing network calls from my EC2 instance are blocked. I can not even perform commands like "apt-get update" or clone repo from github. They get timed out. Also, my server code can not reach AWS S3 resources. Connection to RDS is working fine.
Earlier it was working fine for last few months. Yesterday, I was installing nginx and played with ufw and that is when I noticed the issue and I thought I messed up iptables and thus purged ufw package and flushed iptable rules, rebooted. Still it did not help.
I launched another EC2 instance with the same subnet and other rules, just to see if the issue is with the earlier EC2 instance. Even the new instance is not able to reach external websites. No idea how to investigate further.
UPDATE: Just for completeness posting iptable of my EC2
ubuntu@ip-10-0-1-126:~$ sudo iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT