Some of the nodes in my cluster disappeared just now according to Kubernetes Dashboard and kubectl. In the Google Console, it shows the old amount. This results in many pods not being schedulable due to missing resources.

I tried to increase the node pools now, which is taking much longer than normal, and some new nodes were added, but the amount is still wrong.

Also, in the preemptible pool, no new nodes were added.

What can I do now?

Node Pools according to Google Cloud Console

Only one node in Kubernetes Dashboard

Only one node in kubectl

  • We are currently experiencing the same issue in our production cluster. It seems as if all preemtible nodes are gone. – Sebastian Rösch Oct 25 '18 at 8:19
  • In the preemtible VMs documentation, it is stated that "Preemptible instances are finite Compute Engine resources, so they might not always be available." I get that they can randomly restart, which our cluster can tolerate without a problem. But so far, new nodes were available within minutes. Now, the preemptible VMs have been gone for more than a day and it looks like we're still paying for them. Also, shouldn't there be a warning when no preemptible VMs are available? – Sebastian Rösch Oct 25 '18 at 9:00

It sounds like you are running into a quota or limit issue. I would start checking that:

https://cloud.google.com/compute/quotas https://cloud.google.com/kubernetes-engine/quotas

| improve this answer | |
  • Thanks, Marc, I checked but am well below the limit on each resource quota. And the nodes were there before, so they shouldn't just disappear when I hit the quota, I hope. – Sebastian Rösch Oct 24 '18 at 17:21

To assist you further and help investigate your cluster issue to answer all your questions. I would recommend that you open an Issue Trackers as this platform is meant for defect reports.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.