I'm trying to setup local DNS forwarder for the VPC network to use LDAP controller which is running DNS server. I have few GCP projects which should be able to communicate over already built VPN tunnel to each other using DNS names. I've activated Cloud DNS on GCP, created DNS server policy in order to forward reuqest to the internal DNS servers, assigned this policy to my custom network setup, but GCE VM's are still unable to resolve hosts. But if I will edit /etc/resolv.conf file on the VM and will put nameserver option before the GCP metadata server (169.254.169.254) - everything is working fine. Such sollution isn't preferable as in case of huge amount of VM I'll need to deploy those changes for each VM separately. Also I've tried to deploy forwarding rules for internal domains - result is the same. Accordingly to the GCP DNS "how to" you can use next command in order to check DNS resolver settings for all network:
gcloud compute addresses list \ --filter="purpose=DNS_RESOLVER" \ --format='csv[no-heading](address, subnetwork)' .... 192.168.14.4,cloud-vpn-14 .... This IP was reserved by "dns-forwarder-...." and I'm able to make a request using dig test.1.com but it doesn't forward query to the DNS server which was used by the DNS forwarding policy.
So my question is how to overcome manual override of the /etc/resolv.conf file? Or how to make DNS forwarding working correctly?