I have a Kubernetes cluster on AWS set up with KubeAWS. We're using some sensitive third-party integrations that require white-listed IP addresses in their firewall.
Now, what I've done is that I've registered the IP addresses of the nodes with the external services and added a node label (
whitelisted-ip=true) to those nodes. Services that need to contact those third party providers then have this set as a
nodeSelector. So far so good.
Now, the problem I run into is when the cluster starts scaling, in particular, when it's scaling down. The cluster doesn't seem to care which node it removes, so If I have two nodes in one availability zone, it might as well remove the one which has the white-listed IP, which is then released.
Can I instruct my cluster to, when scaling down, first remove nodes that don't have the node label