I've got two identical GKE clusters (lets call them "blue" and "green"). They each have a single ingress via a GCP Load Balancer. I'd like to trickle traffic from one cluster to the other. This would be used for testing major changes like K8s/Istio upgrades etc.

I'm not sure how I can achieve this with GCP infrastructure.

Options I've explored:

Cloud DNS - this is the simplest, I just switch the A record from the "blue" -> "green" LB and wait for the TTL to expire. This is OK but traffic will take a while to train, and misbehaving DNS clients can potentially cause problems. I also can't trickle traffic across as there is no "weighted round-robin" in Cloud DNS, the switch is "all or nothing" which is not what I want.

Cloud CDN - I can add two LBs as origins, but switching between them would be very slow, and it seems to me a misuse of the tool.

Multi-Layered LBs - Have a LB that sits on top of the "blue" and "green" LBs and directs traffic to the currently "active" one. This doesn't seem possible in GCP at all. Seems LBs can't use LBs as backends (including HTTP, TCP, Network)

Custom top-level LB - I could use HAProxy or something else to front the GCP load balancers, but this makes me nervous. This will be quite a high throughput link and I don't want to compromise on good performance and HA characteristics.

How would I achieve this in GCP without losing the HA and performance properties of the existing setup?

  • The correct procedure is to use Managed Instance Groups (MIG). Create a new MIG for the new traffic. Set balancing mode based up how you want to distribute traffic. Sep 16 '19 at 19:05
  • Do you know if MIGs can be used with GKE? As it seems that GKE manages the creation of instance groups and their registration within GLBs.
    – Boop
    Sep 16 '19 at 21:57
  • 1/2: I am sorry, I overlooked the Kubernetes part. No, MIGs are used for Compute Engine instances. Kubernetes has its own method of load balancing. However, you specified two clusters which means K8s cannot manage traffic. With a single cluster Istio can provide load balancing across multiple versions on the same cluster. Sep 16 '19 at 22:09
  • 2/2: Your first three options are non-starters in GCP. Setting up Nginx or HAProxy will work. Bandwidth will not be a problem unless you are talking 5 or 10 Gigabit / sec or higher. The issue is fault-tolerance unless you build your own proxy cluster. I would use one K8s cluster for development. Then canary deploy with the other K8s cluster and not try to mix them together under a proxy. Sep 16 '19 at 22:17
  • Thanks for your help @JohnHanley. Your points all make sense - I did some research and MIGs actually led me to the link at the bottom, which details a newer feature of GKE, exposing containers as Network Endpoint Groups after all (which can be backends for GLBs). This means they could possibly be TCP load balanced or plugged into GCP's new "Traffic Director" product (effectively Istio outside of the cluster) to achieve the same effect. Will try it out! cloud.google.com/kubernetes-engine/docs/how-to/…
    – Boop
    Sep 17 '19 at 13:49

Actually there are several different ways to load balance the traffic between two GKE clusters.

  • You can leverage the tools: Mutli-cluster Ingress 1 that Google launched. Which will create a Cloud http load balancer for multi-GKE clusters.

  • As you mentioned, you can use Google traffic director 2 to direct traffic to different GKE clusters.

  • And Istio also provides Multiple clusters 3 option, to give you more control of the traffic between multiple Kubernetes clusters.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.