I've got two identical GKE clusters (lets call them "blue" and "green"). They each have a single ingress via a GCP Load Balancer. I'd like to trickle traffic from one cluster to the other. This would be used for testing major changes like K8s/Istio upgrades etc.
I'm not sure how I can achieve this with GCP infrastructure.
Options I've explored:
Cloud DNS - this is the simplest, I just switch the A record from the "blue" -> "green" LB and wait for the TTL to expire. This is OK but traffic will take a while to train, and misbehaving DNS clients can potentially cause problems. I also can't trickle traffic across as there is no "weighted round-robin" in Cloud DNS, the switch is "all or nothing" which is not what I want.
Cloud CDN - I can add two LBs as origins, but switching between them would be very slow, and it seems to me a misuse of the tool.
Multi-Layered LBs - Have a LB that sits on top of the "blue" and "green" LBs and directs traffic to the currently "active" one. This doesn't seem possible in GCP at all. Seems LBs can't use LBs as backends (including HTTP, TCP, Network)
Custom top-level LB - I could use HAProxy or something else to front the GCP load balancers, but this makes me nervous. This will be quite a high throughput link and I don't want to compromise on good performance and HA characteristics.
How would I achieve this in GCP without losing the HA and performance properties of the existing setup?