I would like to find the number of seconds between when the pod was assigned to when the service becomes ready. When i look at the event log for a pod, there is no event that signals when the service becomes ready. Is this something that I need to write a custom script for, or is there somewhere else I can look?
While there is no exact mechanism for that purpose built in Kubernetes provides pod conditions:
A Pod has a PodStatus, which has an array of PodConditions through which the Pod has or has not passed:
PodScheduled: the Pod has been scheduled to a node.
ContainersReady: all containers in the Pod are ready.
Initialized: all init containers have started successfully.
Ready: the Pod is able to serve requests and should be added to the load balancing pools of all matching Services.
With already created Pod you can fetch this data using:
kubectl get pod <pod_name> -oyaml
And observe those conditions:
status: conditions: - lastProbeTime: null lastTransitionTime: "2020-11-06T09:33:30Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2020-11-06T09:33:33Z" status: "True" type: Ready - lastProbeTime: null lastTransitionTime: "2020-11-06T09:33:33Z" status: "True" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2020-11-06T09:33:30Z" status: "True" type: PodScheduled
You can also use
jq for yaml parsing.
Another solution that comes to my mind is Kube-state-metrics:
kube-state-metrics is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects. (See examples in the Metrics section below.) It is not focused on the health of the individual Kubernetes components, but rather on the health of the various objects inside, such as deployments, nodes and pods.
With it you can export object creation time (
kube_pod_start_time) and object ready status (
kube_pod_status_ready). There are of course more metrics available for the pod and other kubernetes objects here.
Please remember that if you are about to measure your
pod start time you have assume that all images needed to run that pod are already pre-pulled on the machine. Otherwise your measurement will not be correct as it will include variables that influence Kubernetes performance such as network or image size.