I had a pod created by a deployment running on a preemptible node in a Google Kubernetes Engine cluster. The node was preempted and recreated. There were several FailedCreatePodSandBox events complaining:
network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
The above events seem to be transient until Calico networking was fully running on the node. However, The final event entry mentioned by "kubectl describe" is different:
Warning FailedCreatePodSandBox 95s (x3 over 101s) kubelet, (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a1afa2a61b7b2260997f4b4719b9c315698d0016af47902923c78e4594e0dc6b" network for pod "pod_name": NetworkPlugin cni failed to set up pod "pod_name" network: Pod "pod_name" is invalid: spec: Forbidden: pod updates may not change fields other than
spec.tolerations(only additions to existing tolerations)
The final event included the pod's entire specification in JSON. The pod remained in the ContainerCreating state for hours, so I assumed it would never recover. I then manually deleted the pod and the deployment immediately created a new one which started quickly on the same node. Did something in the pod specification need to change for the recreated node?
I tried to simulate a preemption by resetting the node, but the pod came right back up in that case. It seems that though the node name stays the same in both cases, there must be some essential difference between recreating a preempted instance and resetting the instance without recreating it.
It seems I encountered a bug, but I'm not sure if it's in Kubernetes proper, GKE's version of Kubernetes, or if it's something specific to Google Cloud Platform's preemption. I'm apparently not the only person to have this problem, since https://github.com/GoogleCloudPlatform/k8s-node-termination-handler exists. I am now using k8s-node-termination-handler and it does work around the problem. Perhaps it's filling in a gap in the functionality GKE provides?