3

I'm getting a "Failed to pull image" error on my deployment in minikube with a local registry, despite other deployments using the same image registry working as expected.

My workflow is to build the image from a Dockerfile called docker.collection as below.

$ minikube start --insecure-registry localhost:5000 --memory 4096
$ docker build . -f docker.collection -t localhost:5000/collection:latest -t localhost:5000/collection:dev
$ docker push localhost:5000/collection
$ cat deployment.yaml | kubectl apply -f -

The deployment is created as desired, but the pods don't start, and the minikube dashboard shows:

Failed to pull image "localhost:5000/collection:dev": rpc error: code = Unknown desc = Error response from daemon: manifest for localhost:5000/collection:dev not found

I have no reason to believe there was any problem in building or pushing the image, and on the host machine, I can pull successfully...

$ docker pull localhost:5000/collection:dev
dev: Pulling from collection
Digest: sha256:8becfdd412f0b86ece8335aa5ee1aede75992d17f88739a28da5939eab28fde5
Status: Image is up to date for localhost:5000/collection:dev

What should else I be checking? How can I get more detail on the failure?

Further debugging

I've also checked:

  • the minikube node can reach localhost:5000:
$ minikube ssh
$ ping localhost:5000
PING localhost:5000 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: seq=0 ttl=64 time=0.126 ms
64 bytes from 127.0.0.1: seq=1 ttl=64 time=0.041 ms
64 bytes from 127.0.0.1: seq=2 ttl=64 time=0.058 ms
^C
--- localhost:5000 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.041/0.075/0.126 ms
$ 
  • a minimal run of the image fails in the same way: kubectl run coll-test --image=localhost:5000/collection:dev yields the same events:
Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  24s               default-scheduler  Successfully assigned dev/coll-test-cf4467b8d-8vpdk to minikube
  Normal   BackOff    23s               kubelet, minikube  Back-off pulling image "localhost:5000/collection:dev"
  Warning  Failed     23s               kubelet, minikube  Error: ImagePullBackOff
  Normal   Pulling    7s (x2 over 23s)  kubelet, minikube  Pulling image "localhost:5000/collection:dev"
  Warning  Failed     7s (x2 over 23s)  kubelet, minikube  Failed to pull image "localhost:5000/collection:dev": rpc error: code = Unknown desc = Error response from daemon: manifest for localhost:5000/collection:dev not found
  Warning  Failed     7s (x2 over 23s)  kubelet, minikube  Error: ErrImagePull
  • the entries for the image exist in the repo:
$ curl http://localhost:5000/v2/_catalog
{"repositories":["collection","foo","bar"]}

Deployment and Dockerfile

# deployment.yaml
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  namespace: dev
  name: collection
  labels:
    app: collection
spec:
  replicas: 1
  selector:
    matchLabels:
      app: collection
  template:
    metadata:
      labels:
        app: collection
    spec:
      containers:
      - name: collection
        image: "localhost:5000/collection:dev"
        command: ["bash", "./collection_entry.sh"]
        imagePullPolicy: Always
# docker.collection
FROM python:3

WORKDIR /code
COPY . /code/
RUN adduser --system --group --no-create-home celery
RUN mkdir /var/run/celery
RUN mkdir /var/log/celery
RUN chown celery:celery /var/run/celery
RUN chown celery:celery /var/log/celery
RUN chmod 755 /var/run/celery
RUN chmod 755 /var/log/celery

RUN pip3 -q install -r requirements.txt
USER celery

ENTRYPOINT ./collection_entry.sh
2

1) You have to run eval $(minikube docker-env)

2) Build the image with the Docker daemon of Minikube

docker build -t collection .

3) Set the image in the pod spec like the build tag - collection

4) Set the imagePullPolicy to Never, otherwise Kubernetes will try to download the image.

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  namespace: dev
  name: collection
  labels:
    app: collection
spec:
  replicas: 1
  selector:
    matchLabels:
      app: collection
  template:
    metadata:
      labels:
        app: collection
    spec:
      containers:
      - name: collection
        image: "collection"
        command: ["bash", "./collection_entry.sh"]
        imagePullPolicy: Never

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.