2

I had a pod created by a deployment running on a preemptible node in a Google Kubernetes Engine cluster. The node was preempted and recreated. There were several FailedCreatePodSandBox events complaining:

network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

The above events seem to be transient until Calico networking was fully running on the node. However, The final event entry mentioned by "kubectl describe" is different:

Warning FailedCreatePodSandBox 95s (x3 over 101s) kubelet, (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a1afa2a61b7b2260997f4b4719b9c315698d0016af47902923c78e4594e0dc6b" network for pod "pod_name": NetworkPlugin cni failed to set up pod "pod_name" network: Pod "pod_name" is invalid: spec: Forbidden: pod updates may not change fields other than spec.containers[*].image, spec.initContainers[*].image, spec.activeDeadlineSeconds or spec.tolerations (only additions to existing tolerations)

The final event included the pod's entire specification in JSON. The pod remained in the ContainerCreating state for hours, so I assumed it would never recover. I then manually deleted the pod and the deployment immediately created a new one which started quickly on the same node. Did something in the pod specification need to change for the recreated node?

I tried to simulate a preemption by resetting the node, but the pod came right back up in that case. It seems that though the node name stays the same in both cases, there must be some essential difference between recreating a preempted instance and resetting the instance without recreating it.

It seems I encountered a bug, but I'm not sure if it's in Kubernetes proper, GKE's version of Kubernetes, or if it's something specific to Google Cloud Platform's preemption. I'm apparently not the only person to have this problem, since https://github.com/GoogleCloudPlatform/k8s-node-termination-handler exists. I am now using k8s-node-termination-handler and it does work around the problem. Perhaps it's filling in a gap in the functionality GKE provides?

1
  • For what it's worth, I've run a GKE cluster with preemptible nodes for most of the past year, tracking the latest stable channel, and haven't encountered this issue Sep 9, 2019 at 20:37

1 Answer 1

1

The issue experienced may rest outside of GKE and with Kubernetes itself. In the Kubernetes github issue page, we see “https://github.com/kubernetes/kubernetes/issues/62666#issue-314798540” which pertains to Pod deletion but has a similar error your received :

 “ spec: Forbidden: pod updates may not change fields other than `spec.containers[].image`, `spec.initContainers[].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)" 

If your experience a similar issue again, I’ll recommend opening a Private Issue here in here, and provide full error you might be getting, GKE version, etc.

By the way, when a node gets preempted, the nodes gets deleted and recreated, so a reset could not have the same impact.

1
  • Thanks for the reply. I will open a private issue if I encounter this problem again. Do you know what state is preserved over a node reset that is not preserved after a preempted node is recreated? Of course the pod definition exists outside the node. I don't think the kubelet preserves any state over a restart, so the only place I can think of state being preserved is the docker daemon. Mar 9, 2019 at 6:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .