Docker private registry as kubernetes pod - deleted images auto-recreated

Question

I run a docker private registry v2.7.0 as a kubernetes pod with a service and a persistent volume, thanks to the Varun Kumar G tutorial, which has been the only successful method on my setup, for kubernetes to pull from my private docker registry on my 3 node--on-premises--cluster with ubuntu 20.04 lts kvms.

The problem is with deleting images from the kubernetes pod docker registry v2.7.0 (had to use the previous version because latest v2.7.1 does not work with htpasswd). Furthermore I have read lots of similar threads like this, this and this.

With docker registry v2.7.1 run as a docker container, I had no problems deleting images,

but with docker registry v2.7.0 run as a kubernetes pod, the usual deletion steps result being unable to push the deleted image again, even after successfully deleting blobs and manually deleting image folders under /var/lib/registry/docker/registry/v2/repositories/.

Below is the registry pod yaml

apiVersion: v1
kind: Pod
metadata:
  name: dockreg-pod
  labels:
    app: mregistry
spec:
  containers:
  - name: registry
    image: registry:2.7.0
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: repo-vol
      mountPath: "/var/lib/registry"
    - name: certs-vol
      mountPath: "/certs"
      readOnly: true
    - name: auth-vol
      mountPath: "/auth"
      readOnly: true
    env:
    - name: REGISTRY_AUTH
      value: "htpasswd"
    - name: REGISTRY_AUTH_HTPASSWD_REALM
      value: "Registry Realm"
    - name: REGISTRY_AUTH_HTPASSWD_PATH
      value: "/auth/htpasswd"
    - name: REGISTRY_HTTP_TLS_CERTIFICATE
      value: "/certs/tls.crt"
    - name: REGISTRY_HTTP_TLS_KEY
      value: "/certs/tls.key"
    - name: REGISTRY_STORAGE_DELETE_ENABLED
      value: "true"
  volumes:
  - name: repo-vol
    persistentVolumeClaim:
      claimName: repo-pvc
  - name: certs-vol
    secret:
      secretName: certs-secret
  - name: auth-vol
    secret:
      secretName: auth-secret
  restartPolicy: Always
  nodeName: spring

and following is the persistent volume yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: repo-pv
  labels:
    type: prstore
spec:
  capacity:
    storage: 7Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    fsType: ext4
    path: /root/repo
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - spring
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: repo-pvc
  labels:
    type: prstore
spec:
  selector:
    matchLabels: 
      type: prstore
  volumeMode: Filesystem
  storageClassName: local-storage
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 7Gi

Say I am pushing an image on a brand new registry pod having also wiped out the persistent storage beforehand.

root@sea:scripts# docker push dockreg:5000/mubu4:v4
The push refers to repository [dockreg:5000/mubu4]
9f54eef41275: Pushed 
v4: digest: sha256:7bd0d9a9821815dccb5c53c18cea04591ec633e2e529c5cdd39681169589c17f size: 529

Deletion of images appears to work as intended until I try to push the deleted image again, at which point I am getting the dreaded Layer already exists error.

As you might have seen above, I have included in the registry pod environment the following,

- name: REGISTRY_STORAGE_DELETE_ENABLED
      value: "true"

otherwise I would get an unsupported error from the curl -X DELETE call, even after adding

delete:
    enabled: true

in the /etc/docker/registry/config.yml within the pod,

version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
  delete:
    enabled: true
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3

that seems to make no difference in my use case.

Following are the deletion steps.

curl -u alexander:sofianos \
> -vsk -H "Accept: \
> application/vnd.docker.distribution.manifest.v2+json" \
> -X DELETE \
> https://dockreg:5000/v2/mubu4/manifests/sha256:\
> 7bd0d9a9821815dccb5c53c18cea04591ec633e2e529c5cdd39681169589c17f

The above prints among other things the following

> DELETE /v2/mubu4/manifests/sha256:7bd0d9a9821815dccb5c53c18cea04591ec633e2e529c5cdd39681169589c17f HTTP/2
> Host: dockreg:5000
> authorization: Basic YWxleGFuZGVyOnNvZmlhbm9z
> user-agent: curl/7.68.0
> accept: application/vnd.docker.distribution.manifest.v2+json
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 202 
< docker-distribution-api-version: registry/2.0
< x-content-type-options: nosniff
< content-length: 0
< date: Sat, 30 Oct 2021 13:25:53 GMT
< 
* Connection #0 to host dockreg left intact

which seems to be in order.

Below, deleting blobs from within the registry pod

root@sea:scripts# kubectl exec -it dockreg-pod -- sh
/ # bin/registry garbage-collect  /etc/docker/registry/config.yml
mubu4

0 blobs marked, 3 blobs and 0 manifests eligible for deletion
blob eligible for deletion: sha256:7b1a6ab2e44dbac178598dabe7cff59bd67233dba0b27e4fbd1f9d4b3c877a54
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/7b/7b1a6ab2e44dbac178598dabe7cff59bd67233dba0b27e4fbd1f9d4b3c877a54  go.version=go1.11.2 instance.id=82a101ee-47f4-4f4f-bc79-76d774b0924b service=registry
blob eligible for deletion: sha256:7bd0d9a9821815dccb5c53c18cea04591ec633e2e529c5cdd39681169589c17f
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/7b/7bd0d9a9821815dccb5c53c18cea04591ec633e2e529c5cdd39681169589c17f  go.version=go1.11.2 instance.id=82a101ee-47f4-4f4f-bc79-76d774b0924b service=registry
blob eligible for deletion: sha256:ecb35fc8715f5ab1d9053ecb2f2d9ebbec4a59c0a0615d98de53bc29f7285085
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/ec/ecb35fc8715f5ab1d9053ecb2f2d9ebbec4a59c0a0615d98de53bc29f7285085  go.version=go1.11.2 instance.id=82a101ee-47f4-4f4f-bc79-76d774b0924b service=registry

Lastly, manually deleting the repository image

/ # rm -rf /var/lib/registry/docker/registry/v2/repositories/mubu4

On my persistent storage, the registry now looks like this

root@spring:repo# tree
.
└── docker
    └── registry
        └── v2
            ├── blobs
            │   └── sha256
            │       ├── 7b
            │       └── ec
            └── repositories

8 directories, 0 files

But when I try to push the deleted image again, I get

root@sea:scripts# docker push dockreg:5000/mubu4:v4
The push refers to repository [dockreg:5000/mubu4]
9f54eef41275: Layer already exists 
v4: digest: sha256:7bd0d9a9821815dccb5c53c18cea04591ec633e2e529c5cdd39681169589c17f size: 529

and in my registry, the mubu4 image folder I previously deleted, has been mystiriously recreated through the above push command.

root@spring:repo# tree
.
└── docker
    └── registry
        └── v2
            ├── blobs
            │   └── sha256
            │       ├── 7b
            │       └── ec
            └── repositories
                └── mubu4
                    └── _manifests
                        ├── revisions
                        │   └── sha256
                        │       └── 7bd0d9a9821815dccb5c53c18cea04591ec633e2e529c5cdd39681169589c17f
                        │           └── link
                        └── tags
                            └── v4
                                ├── current
                                │   └── link
                                └── index
                                    └── sha256
                                        └── 7bd0d9a9821815dccb5c53c18cea04591ec633e2e529c5cdd39681169589c17f
                                            └── link

19 directories, 3 files

I also tried wiping out the persistent storage with

root@spring:repo# rm -rf *

to no avail. Trying to push the deleted image afterwards, still outputs the exact same Layer already exists error, and the registry tree is being again auto-recreated, looking exactly as it does in the above tree output.

The question is what else can I try to make this work, and/or alternatively,

it follows from the above testing, that within the docker registry kubernetes pod, there are other files, that hold configuration where deleted images appear not to be deleted, and these files activate the recreation of the deleted image through a docker push call. Where should I look apart from the tree

/var/lib/registry/docker/registry/v2/

so I can delete all references to deleted images?

Answer 1

It appears that there is a caching issue at least with registry version 2.7.0 that is described here and here.

It is recommended in the above threads to disable cache completely or to restart the container every time.

Because I am using the docker registry as a kubernetes pod, changes to the default registry configuration file, ie. /etc/docker/registry/config.yml do not have any effect because the kubernetes registry pod yaml takes precedence, meaning the configuration has to be set in the pod yaml as environment variables in the form of REGISTRY_variable where the underscore represents indentation levels, as explained in the docs.

So the solution is to add

- name: REGISTRY_STORAGE_CACHE_BLOBDESCRIPTOR
  value: ""

to the container environment in the pod yaml in order to disable cache completely, otherwise if the registry is run as a docker container we can use the following in the config.yml:

storage:
  cache:
    blobdescriptor: ""

The alternative is to restart the pod every time we delete an image with:

kubectl exec <pod_name> -c <container_name> -- reboot

or if it is a docker container

docker restart <registry_container>