We are running a Google Kubernetes Engine cluster where all the nodes are marked "preemptible". From the Google documentation:
Preemptible VMs are Compute Engine VM instances that last a maximum of 24 hours and provide no availability guarantees.
However, when I look at my pods running on one of these nodes I see this:
NAME READY STATUS RESTARTS AGE mypod-dev-0 3/3 Running 0 20h mypod-dev-1 3/3 Running 0 26h
Note that the age of mypod-dev-1 is more than 24 hours. Using
pod describe on mypod-dev-1 I see that all the containers were started at 07:08 AM this morning (about 3 hours ago).
Turning to the node that mypod-dev-1 is running on, when I look at the node details I see some things that are very confusing. First of all, the creation time is more than 24 hours ago (current time 10:00 AM on 20 Dec 2019):
CreationTimestamp: Thu, 19 Dec 2019 06:55:26 -0800
Next, there are a bunch of "Conditions" messages that suggest that the node was re-created more recently:
Type Status LastHeartbeatTime LastTransitionTime Reason Message NetworkUnavailable False Thu, 19 Dec 2019 06:55:26 -0800 Thu, 19 Dec 2019 06:55:26 -0800 RouteCreated NodeController create implicit route KernelDeadlock False Fri, 20 Dec 2019 10:03:00 -0800 Fri, 20 Dec 2019 07:07:21 -0800 KernelHasNoDeadlock kernel has no deadlock ReadonlyFilesystem False Fri, 20 Dec 2019 10:03:00 -0800 Fri, 20 Dec 2019 07:07:21 -0800 FilesystemIsNotReadOnly Filesystem is not read-only ...
It appears that mypod-dev-1 had all of its containers restarted at the same time (07:08 AM) something happened with the node.
How do I determine when a node was created?
Why does the pod time show an age longer than what is supposed to be allowed by the preemptible nature of the nodes?
Is there some log that shows when a pod was migrated off of one node and on to another?