We have a deployment of security service on Kubernetes that needs to have the same private key to encrypt and decrypt API tokens that we send to clients.
At the moment, all the pods need to have the same private key. For example:
- Pod A issues an API key to Alice
- Alice then wants to access a secure resource
- Alice makes the request that gets routed to pod B
- If pod B doesn't have the same private key as pod A, then pod B will be unable to decrypt the key to verify, meaning Alice erroneously gets locked out
We'd like to find a way in Kubernetes to condition pod liveness on that pod having the same private key as the other pods. I realize this could create a death spiral situation if this fails, but that's fine. We do a blue-green deployment, so the live traffic would never get switched to the degenerate deployment until it was up and running.
My current thought is to write a script that runs as a liveness probe that compares the checksum of the private key file against the checksums of all the other pods, pulling the list of sibling pods from the Kubernetes internal APIs.
This is motivated by a case where the image pull policy was set to
IfNotPresent and then an image with the same tag was updated between pod deployments, so we wound up breaking the service because different private keys were in different places. Obviously we could add checking this to our deployment checklist, but if there's a way to automate the enforcement of the policy through the actual deploy tool, I feel like that would be stronger.
I'm also willing to believe this approach of having multiple pods that all need to have the same key is the wrong approach for some reason. If there's a better way to achieve this goal, that would also be a useful answer to this question.