We have a new Google Cloud Platform setup. We have successfully created a VPN connection to our office router. We can reach private addresses in Google Cloud from our office and some services from inside GCP can reach into our office network. We have set up kube-dns to use our internal DNS server over the VPN for private resources and for the most part, that's working fine. And, from both containers and the container cluster node VMs, we can ping addresses inside the office network.
But the first app we wish to deploy in Google Container Engine needs HTTPS access to API servers running inside the office network behind the firewall. These HTTPS connections (tested with wget and curl) just hang. No data is returned. I've done a traffic capture on the office router and I can see packets coming in and going back out the tunnel as expected. I also tried regular HTTP and SSH connetions outbound from Google into our office. I tried from inside a running container and from the Container Engine cluster node (COS image) itself. Nothing but ping and DNS are working from Google reaching into the office network.
There are no Egress firewall rules in GCP. The office router policy is presently allowing all traffic across the VPN. I've found a couple of other posts similar to our situation, but they were older and most of them reference fixes to GCP that have been made in the meantime, others discuss complex iptables solutions.
So, we are totally blocked on our first cloud deployment. How can we troubleshoot the networking in GCP? It seems like the returning packets are just not getting routed back to their source. Neither DNS nor ping are TCP and of course HTTPS and SSH are TCP. That's the main difference I can identify between what works and what does not. What do we try next?