2

OK, so I have a VPC with three app servers and an instance of Postgres in RDS.

I have a security group called 'rds-staging' that allows inbound connections on port 5432 from a security group called 'app-elb-staging'.

'app-elb-staging' is the security group applied to all of my EC2 instances, and it allows outgoing traffic to go anywhere.

The RDS instance is in AZ us-east-1e. I can connect to it from my EC2 instance in us-east-1e (10.0.3.*), but not from any EC2 instances in us-east-1a (10.0.1.*) or us-east-1c (10.0.2.*):

deploy@ip-10-0-3-220:~$ nc -zv xxx.us-east-1.rds.amazonaws.com 5432
Connection to xxx.us-east-1.rds.amazonaws.com 5432 port [tcp/postgresql] succeeded!

deploy@ip-10-0-1-155:~$ nc -zv xxx.us-east-1.rds.amazonaws.com 5432
nc: connect to xxx.us-east-1.rds.amazonaws.com port 5432 (tcp) failed: No route to host

deploy@ip-10-0-2-90:~$ nc -zv xxx.us-east-1.rds.amazonaws.com 5432
nc: connect to xxx.us-east-1.rds.amazonaws.com port 5432 (tcp) failed: No route to host

Has anyone seen this before? I've checked the DNS, and each machine is resolving the hostname to the same IP (10.0.3.x).

  • "each machine is resolving the hostname to the same IP" ... Is it a 10.x.x.x IP? Or a public IP? – Michael - sqlbot Jul 23 '16 at 13:07
  • @Michael-sqlbot the DB hostname resolves to a 10.0.3.x IP – Codebeef Jul 23 '16 at 16:52
  • @Michael-sqlbot the DB instance is not publicly accessible :) – Codebeef Jul 23 '16 at 16:56
2

OK, finally figured the root cause of this issue. The AMI I was using was creating a bridge that caused the connection issue due to it colliding with the IPs of my subnets. The output from sudo route -n looked like this on an affected instance:

ubuntu@ip-10-0-1-92:~$ sudo route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.1.1        0.0.0.0         UG    0      0        0 eth0
10.0.1.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
10.0.2.0        0.0.0.0         255.255.255.0   U     0      0        0 lxcbr0

Any connection to 10.0.2.* would then fail:

deploy@ip-10-0-1-92:~$ nc -zv 10.0.2.53 22
nc: connect to 10.0.2.53 port 22 (tcp) failed: No route to host

Removing the bridge with sudo ifconfig lxcbr0 down resolved the issue, but using an AMI that does not set this bridge up in the first place corrected the root.

1

I've seen this kind of problem caused by one of two reasons:

  1. The route-tables attached to the various subnets are configured wrong.
  2. There is a Network ACL in play (unusual).

You don't need to define routes for each subnet, they're implicit in the table. Double-checking the IP addresses that your DNS entry resolves to on the other-AZ instances will ensure that it is in the VPC.

Network ACLs can come into play, but you have to set them up. By default, they're wide open. This is why I tagged it as unlikely, but it can cause problems like this. That said, the 'no route to host' error is suggestive that this isn't your problem.

  • Have had a look at my route tables, but they seem fine from what I can gather. Each subnet is attached to the same route table, which has two routes: 10.0.0.0/16 - local and 0.0.0.0/0 - routed to the nat instance. The network ACLs also seem fine - wide open, as you mention. – Codebeef Jul 24 '16 at 16:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.