I have a running cluster on Kubernetes with GCP, and some services running on App Engines, and I'm trying to communicate between them without being able to access the App Engines from the outside.
I created a private Kubernetes cluster with a specific subnetwork, I linked this subnetwork to a Cloud NAT to have a unique egress IP I can whitelist, and I allowed this IP in the App Engine firewall rules.
However, when I request my app engines from the cluster, I get a 403 response because it doesn't pass through the firewall. But if I connect to my Kubernetes pod and try to request a site to know my IP, I get the IP I set in the Cloud NAT.
I found in the Cloud NAT documentation that the translation to internal IPs is realised before the application of the firewall rules (https://cloud.google.com/nat/docs/overview#firewall_rules).
When I connect to the app engines, I can see that requests that are made through the NAT gateway come in with IP 0.0.0.0. However, when we whitelist only this ip, the requests goest through only 1 out of 3 times, which seems really weird.
Would you know how to understand and fix this?