We're running across three zones in GCE with a TCP Proxy loadbalancer in front. The backend for the loadbalancer is on a single node within one of the zones. Using the health check, the loadbalancer can determine where to send the traffic. This works as expected when traffic originates outside of Google. For outgoing traffic we use a NAT gateway and a route with a lower priority than the default Google route (800 instead of 1000) to actually route the traffic for 0.0.0.0/0 there.
However, when one of the backend servers that's not in the same zone as the (active) backend for the TCP loadbalancer tries to access the external address, it will get a connection time out. The route does not seem to get used. If the request originates from the same zone as the active backend, it works. The nat gateway route does not seem to be used.
One solution would of course be to make sure that there's an active backend in all zones, but we prefer not to do that. One thing we note is that when we get traffic from within the cluster through this external IP address, the client IP address in our logs seems to be the loadbalancer address itself. Any idea how to solve this otherwise? Is this expected behaviour or did we possibly misconfigure some route or firewall?