tl;dr We're experiencing issues with services behind internal Google Cloud Load Balancer not being able to reply when response is over a certain size. Upon some tcpdump'ing and wireshark'ing, it looks like our site-to-site ipsec VPN instance cannot send a ICMP "fragmentation needed" packet back to GCLB, causing the response to be stuck in a TCP retransmission loop.
We have a site-to-site IPSec tunnel setup for a GCP VPC and our onsite gateway. We have some vm instances in that VPC and we created internal GCLB (Google Cloud Load Balancer) to lb those services. However, we noticed that when the response of the service goes over a certain size, the
curl request to the GCLB (from our internal network) gets stuck and eventually timeout.
I did a tcpdump at our VPN instance (the instance that runs ipsec) in GCP, and captured the trace when the response is "big". It turns out that the VPN tries to send a ICMP fragmentation needed packet back to the internal GCLB but got "no route to host". I verified that there is a route to GCLB from the VPN instance and I can even curl the GCLB instance at the VPN instance. That made me suspect that the firewall is blocking the ICMP packet.
However, playing around with GCP's firewall rules, I haven't been able to make firewall rules apply to GCLB in the network. It seems that all firewall rules apply to VM instances.
Could someone shed some light on how to get around this?