93

A pod in my Kubernetes cluster is stuck on "ContainerCreating" after running a create. How do I see logs for this operation in order to diagnose why it is stuck? kubectl logs doesn't seem to work since the container needs to be in a non-pending state.

116

kubectl describe pods will list all the events associated with the pod, including pulling of images, starting of containers. It might be of help.

  • 5
    what if the container stuck at ContainerCreating without any events? for me the events is shown as "No events." – Bob May 26 '16 at 2:30
  • 1
    Some events seem to take a while to show up. For example a timeout attempting to mount a disk for me takes about 2 minutes before it shows up as an event. – jwadsack Jul 28 '16 at 18:49
  • 7
    It happens when you are using secrets and they are not found (like a typo in the yaml or you forgot to create it before). For the almost all other possible errors it gets CrashLoopback or Error states but with secrets it just gets stuck in ContainerCreating, if you describe the pod then you'll see at the very end a message saying the secret was not found, but it barely says nothing about the problem. – danius Oct 31 '16 at 20:53
  • Yeah usually you don't have any events before he starts doing something. – erikbwork May 10 '17 at 14:44
  • Happened to me this morning and it was a typo in a hostPath for a volume. Yay sticky keyboard. – Joe Block Jan 14 at 17:26
0

The one time I hit this was because my resource declarations were accidentally very very small.

resources: limits: cpu: 1000m memory: 1024M requests: cpu: 1000m memory: 1024M

vs

resources: limits: cpu: 1000m memory: 1024m requests: cpu: 1000m memory: 1024m

capitalizing that m makes a very large difference in resource use. I was stuck on ContainerCreating because I had not given enough memory to my container.

3

In my case, docker's access to internet was blocked. It was solved using a proxy (using sandylss's comment):

  1. minikube stop
  2. minikube delete
  3. export http_proxy=http://user:pass@ip:port
  4. export https_proxy=http://user:pass@ip:port
  5. export no_proxy=192.168.99.0/24
  6. minikube start --logtostderr --v=0 --bootstrapper=localkube --vm-driver hyperv 
      --hyperv-virtual-switch "Primary Virtual Switch" --docker-env HTTP_PROXY=$http_proxy \
      --docker-env HTTPS_PROXY=$https_proxy --docker-env NO_PROXY=$no_proxy
    
  7. export no_proxy=$no_proxy,$(minikube ip)
  8. export NO_PROXY=$no_proxy,$(minikube ip)

Then, to check if docker has access to internet, run:

$ docker pull tutum/hello-world

in the cluster (connect to the cluster using minikube ssh); stop the process if it starts downloading.

My second problem was slow internet connection. Since the required docker images are on the order of 100MB, both docker containers and Kubernetes pods remained in \pause and ContainerCreating states for 30 minutes.

To check if docker is downloading the images, run:

$ ls -l /var/lib/docker/tmp

in the cluster, which shows the temporary image file[s] that are being downloaded, empty otherwise.

If you are developing in minikube and using VPN, docker can use your VPN via fiddler. That is, docker will be connected to fiddler's ip:port, and fiddler is connected to the VPN. Otherwise, VPN is not shared between your host and minikube VM.

  • Got bit by this bug today. Still not sure what caused it though. Things were working fine one minute and the next, this issue cropped up. Thank you for the fix. It worked for me. – Jim Oct 31 '18 at 6:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.