AWS private hosted zone setup

Question

I'm trying to set up a pair of VPC peers on Amazon. One of the VPCs has been in use for some time, and the other is a new VPC I've just set up. As part of this effort, I've associated an existing Route 53 Private Hosted Zone with the new VPC. But for some reason, the EC2 instances in the new VPC are not able to perform any name lookups from the Private Hosted Zone.

I've enabled DNS Resolution and DNS Hostnames for the new VPC. I can't imagine the peering setup is causing issues, but I've setup and accepted the peering connection and configured routes between the new VPC and the existing one. The peering connection seems to be working, as I am able to open a connection to an EC2 instance on the old VPC.

I'm not exactly sure where the disconnect is. I've gone through every option I could find for Privated Hosted Zone and the VPC, but I haven't found anything promising.

---

old VPC

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface 
0.0.0.0         10.0.13.1       0.0.0.0         UG    0      0        0 eth0 
10.0.13.0       0.0.0.0         255.255.255.0   U     0      0        0 eth0
169.254.169.254 0.0.0.0         255.255.255.255 UH    0      0        0 eth0

$ dig sql04 +trace
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.37.rc1.45.amzn1 <<>> sql04 +trace
;; global options: +cmd
.           518400  IN  NS  K.ROOT-SERVERS.NET.
.           518400  IN  NS  L.ROOT-SERVERS.NET.
.           518400  IN  NS  M.ROOT-SERVERS.NET.
.           518400  IN  NS  A.ROOT-SERVERS.NET.
.           518400  IN  NS  B.ROOT-SERVERS.NET.
.           518400  IN  NS  C.ROOT-SERVERS.NET.
.           518400  IN  NS  D.ROOT-SERVERS.NET.
.           518400  IN  NS  E.ROOT-SERVERS.NET.
.           518400  IN  NS  F.ROOT-SERVERS.NET.
.           518400  IN  NS  G.ROOT-SERVERS.NET.
.           518400  IN  NS  H.ROOT-SERVERS.NET.
.           518400  IN  NS  I.ROOT-SERVERS.NET.
.           518400  IN  NS  J.ROOT-SERVERS.NET.
;; Received 496 bytes from 10.0.0.2#53(10.0.0.2) in 100 ms

.           86400   IN  SOA a.root-servers.net. nstld.verisign-grs.com. 2016052600 1800 900 604800 86400
;; Received 98 bytes from 193.0.14.129#53(193.0.14.129) in 77 ms

$ nslookup sql04
Server:     10.0.0.2
Address:    10.0.0.2#53

Non-authoritative answer:
Name:   sql04.domain.local
Address: 10.1.3.28

new VPC

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.1.255.1      0.0.0.0         UG    0      0        0 eth0
10.1.255.0      0.0.0.0         255.255.255.192 U     0      0        0 eth0
169.254.169.254 0.0.0.0         255.255.255.255 UH    0      0        0 eth0

$ dig sql04 +trace

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.37.rc1.45.amzn1 <<>> sql04 +trace
;; global options: +cmd
.           518400  IN  NS  M.ROOT-SERVERS.NET.
.           518400  IN  NS  A.ROOT-SERVERS.NET.
.           518400  IN  NS  B.ROOT-SERVERS.NET.
.           518400  IN  NS  C.ROOT-SERVERS.NET.
.           518400  IN  NS  D.ROOT-SERVERS.NET.
.           518400  IN  NS  E.ROOT-SERVERS.NET.
.           518400  IN  NS  F.ROOT-SERVERS.NET.
.           518400  IN  NS  G.ROOT-SERVERS.NET.
.           518400  IN  NS  H.ROOT-SERVERS.NET.
.           518400  IN  NS  I.ROOT-SERVERS.NET.
.           518400  IN  NS  J.ROOT-SERVERS.NET.
.           518400  IN  NS  K.ROOT-SERVERS.NET.
.           518400  IN  NS  L.ROOT-SERVERS.NET.
;; Received 228 bytes from 10.1.0.2#53(10.1.0.2) in 161 ms

.           86400   IN  SOA a.root-servers.net. nstld.verisign-grs.com. 2016052600 1800 900 604800 86400
;; Received 98 bytes from 192.33.4.12#53(192.33.4.12) in 38 ms

$ nslookup sql04
Server:     10.1.0.2
Address:    10.1.0.2#53

** server can't find sql04: NXDOMAIN

Answer 1

To echo what @Michael-sqlbot wrote in his comment - I encountered a very similar issue here. The problem was my private hosted zone's associated VPC was different. I had a VPC peering connection between two VPCs (let's call them VPC1 and VPC2).

The EC2 instance I was performing my nslookup within was on VPC1, but my associated private hosted zone was with VPC2. Just like the OP's post, nslookup shows that my DNS query is being served by the VPC's default DNS server (the 10.1.0.2 address). However, if you don't associate the private hosted zone with that VPC, then just like the error indicates, the DNS server doesn't know how to resolve that query. Here is the screen in the Route 53 console you perform your associations with (note you can associate multiple VPCs):