What are the reason to consider using one k8s cluster for blue and one for green when doing B/G CD scenario ? Does the cost of having two clusters which already have cloud provider HA has reasoning ?
Nothing to do with HA, it just lets you test new code releases for half of your clients with the other half running known-good code and you then have the ability to flip the new code clients onto the old code if there's a problem. It's that simple.
I would much rather use two distinct deployments like
myapp-green on the same cluster. With proper resource management, or even node selectors / taints / tolerations you can achieve a very high level of controll to make sure that they do not interfere with each other if you need that level of separation. Don't see much gain to maintain two clusters for blue-green releases as main reason.
From experience I can say it's really good to have a separate cluster to run pre-prod tests for a release. You are still work on the live consistent traffic/data, you can loadbalance it for one particular client or data entry, also maybe you're releasing a new feature to be tested by one particular group.
Recently there was a huge fuss about this new EU law for companies called GDPR, which forced you to treat EU costumers/clients personal data differently from others. You could just route the EU traffic to blue cluster which would be in compliance with GDPR.