After migrating all our Kubernetes workloads to private clusters, we've found that the SD metrics explorer is no longer receiving any data. Although we have cloud NAT setup to work with all subnets, pods in kube-system are unable to connect to the Google API server. In fact, they are unable to make outbound TCP connections to anything on the internet. However, they can ping internet addresses. I've been unable to find a firewall rule that would negate the default allow for egress traffic. Looking for any solution or ideas for additional troubleshooting.
From this another serverfault question, you can see that by default Unable to access internet on pod in private GKE cluster private GKE cluster do not have external IP addresses.