I have a production cluster with six nodes (three masters, three workers) built with
kops and hosted on AWS. I have a problem where, when a single pod has heavy load, the entire node seems to lock. However, even though the pod is under strain, there are plenty of resources left on the node.
Despite this, my pods die with a
NodeLost-error and the pod doing some heavy lifting is killed and the work lost.
Why is the node lost even though there's plenty of resources available, and how do I prevent this from happening? I'm expecting load increase on all services this coming year, and I feel I'm missing something important here.
Please see attached images.
Thanks for your time!