I have a production cluster with six nodes (three masters, three workers) built with kops and hosted on AWS. I have a problem where, when a single pod has heavy load, the entire node seems to lock. However, even though the pod is under strain, there are plenty of resources left on the node.

Despite this, my pods die with a NodeLost-error and the pod doing some heavy lifting is killed and the work lost.

Why is the node lost even though there's plenty of resources available, and how do I prevent this from happening? I'm expecting load increase on all services this coming year, and I feel I'm missing something important here.

Please see attached images.

Thanks for your time!

Nodes during the height of the load: Node load

Pods on the node soon after: Pods ("node lost")

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.