We are running private GKE cluster over GCP. Our services expose to the internet via nginx-ingress and TCP LB white listed in the service.yaml definition.
One of our pods is trying to access another pod through the Public LB. (I know it's not the best practice but lets assume the app need to work only this way and we can't handle in cluster communication.)
I've also added the NAT static IP to the LB whitelist and I can see the IP as a firewall rule that connected to the GKE nodes.
What I've tried to debug the issue:
Create new instance, install nginx and allow connections only from the NAT IP by adding firewall rules. I’ve also tried to connect tcp LB to this nginx instance and have no problem to access the nginx sample page from both pod A and B.
When I’m trying to connect to the pods from other whitelisted sources I have no issue at all.