0

tl;dr We're experiencing issues with services behind internal Google Cloud Load Balancer not being able to reply when response is over a certain size. Upon some tcpdump'ing and wireshark'ing, it looks like our site-to-site ipsec VPN instance cannot send a ICMP "fragmentation needed" packet back to GCLB, causing the response to be stuck in a TCP retransmission loop.

Details

We have a site-to-site IPSec tunnel setup for a GCP VPC and our onsite gateway. We have some vm instances in that VPC and we created internal GCLB (Google Cloud Load Balancer) to lb those services. However, we noticed that when the response of the service goes over a certain size, the curl request to the GCLB (from our internal network) gets stuck and eventually timeout.

I did a tcpdump at our VPN instance (the instance that runs ipsec) in GCP, and captured the trace when the response is "big". It turns out that the VPN tries to send a ICMP fragmentation needed packet back to the internal GCLB but got "no route to host". I verified that there is a route to GCLB from the VPN instance and I can even curl the GCLB instance at the VPN instance. That made me suspect that the firewall is blocking the ICMP packet.

See the wireshark screenshot here

However, playing around with GCP's firewall rules, I haven't been able to make firewall rules apply to GCLB in the network. It seems that all firewall rules apply to VM instances.

Could someone shed some light on how to get around this?

0

From the subject it seems that your question is : “Can firewall rules be applied to internal GCLB?” the answer is no. Firewall rules are applied directly to VMs, not load balancers. The internal load balancer is a pass-through load balancer. To control traffic, you can apply firewall rules to the VMs themselves, but you cannot apply them "to the load balancer." You could check this documentation for further information.

After going through your details it seems like the you might have configured a VM to run IPSec software as you mentioned “I did a tcpdump at our VPN instance (the instance that runs ipsec) in GCP.” I would recommend that they use Cloud VPN instead so that you could make sure that things like MTUs are set correctly. There are so many places where the instance could have things like MTU misconfigured.

Note that the maximum MTU in GCP is 1460 bytes, but that the MTU for VPN traffic should be less than to account for packet encapsulation. You could look at the recommendations for on-premises gateways when using Cloud VPN.

The other thing that they should be doing is pre-fragmenting the traffic before sending it over the tunnel. The traffic must be pre-fragmented before it is encapsulated. That's a configuration that's required for the peer device connected to Cloud VPN. Here is the documentation link for that. As it seems you are running a VM based VPN, you need to handle pre-fragmentation on their VM and their peer gateway.

I would like to mention here that Cloud VPN does pre-fragmentation for you, so the only consideration there is the peer device.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.