35

I have a single node kubernetes cluster in google container engine to play around with.

Twice now, a small personal website I host in it has gone offline for a couple minutes. When I view the logs of the container, I see the normal startup sequence recently completed, so I assume a container died (or was killed?) and restarted.

How can I figure out the how & why of this happening?

Is there a way to get an alert whenever a container starts/stops unexpectedly?

50

You can view the last restart logs of a container using:

kubectl logs podname -c containername --previous

As described by Sreekanth, kubectl get pods should show you number of restarts, but you can also run

kubectl describe pod podname

And it will show you events sent by the kubelet to the apiserver about the lifecycled events of the pod.

You can also write a final message to /dev/termination-log, and this will show up as described in the docs.

| improve this answer | |
13

Beside the previous answers another command that helped me to find an error is:

kubectl get event [--namespace=my-namespace]

It lists events from Pods, Jobs, Nodes too

| improve this answer | |
8

kubectl get pods will actually list any restarts of the container also the describe command can be of help cause it lists any events associated with the pod.

Liveness probes and readiness probes can be configured for better handling check here

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

Additionally hooks can be configured to be consumed in the container at specific points in the life cycle of the container check here

https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/

| improve this answer | |
  • 1
    Please expand the links into the answer. Otherwise -1. :) – Konrad Gajewski Oct 22 '15 at 11:24
0

I follow these steps to define the reason for failure:

# print out a pod logs (https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#logs)
kubectl logs {name_of_pod} -n {namespace} --since=2h --timestamps


# print the logs for the _previous_ instance of the container in a pod if it exists
kubectl logs -p {name_of_pod} -n {namespace} --since=2h --timestamps


# check events (https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#describe)
kubectl describe pod {pod_name} -n {namespace}

# look at the 'Events' at the end of the output
# ..
# Events:
#   Type     Reason   Age                 From               Message
#   ----     ------   ----                ----               -------
#   Warning  BackOff  40m                 kubelet, gke-xx    Back-off restarting failed container
# ..


# observe all events (https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get)
kubectl get events -n {namespace} --sort-by=.metadata.creationTimestamp


# check logs, etc. in pod container (https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#exec)
kubectl exec -it {pod_name} -n {namespace} -- sh
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.