We have a 6 node Kubernetes cluster running around 20 large replica set workloads (Java services). Each workload pod (1 pod per workload) takes about 30 seconds on average to start and use a lot of CPU. This makes starting multiple pods/workloads at the same time a problem - to the point that when 2 or 3 start at the same time on the same node they take minutes to start and eventually get killed by the readiness probe. The readiness probe is fairly relaxed, but extending the grace time indefinitely doesn't seem like good practice.
As one can imagine, this makes cordoning and draining a node problematic - if we drain a node all the pods restart at the same time somewhere else and can overload a worker (or bring it to a standstill causing multiple restarts which eventually lead to database locks).
To get around this I've written a shell script which uses kubectl to list out the pods, restart each (by patching the meta data), wait for status to become available and move to the next one.
Scripts work fine for server patching or workload upgrades, but don't solve the problem of a node outage - everything runs in AWS and when a node fails a new one is created via autoscaling, but it means 4 pods try and restart at the same time (usually on Sunday morning at 3am of course).
One idea would be to have an init container which is aware of the other starting workloads - if no other workloads are currently starting on the same node, then the init container exits allowing the main container to start. This would require a service account and permissions, but could be a workaround, but I was wondering if there was a more standard way to do this via configuration (affinity rules etc)?