Getting "x509: certificate signed by unknown authority" in GKE on pulling image (a private registry) when a pod is created

Question

I generated a CA certificate, then issued a certificate based on it for a private registry, that located in the same GKE cluster. Put the server certificates to the private registry and the CA certificate to all GKE nodes and run:

update-ca-certificates && systemctl restart docker

Images are building and putting into the private registry without problems. When a pod tries to pull the an image from the repository I get an error:

x509: certificate signed by unknown authority

Also I tried to put the CA certificate to the docker certs.d directory (10.3.240.100:3000 — the IP address of the private registry) and restart the docker on each node of the GKE cluster, but it doesn't help too:

/etc/docker/certs.d/10.3.240.100:3000/ca.cert

How to solve this problem? Am I understand correctly that the GKE nodes' docker is responsible for pulling images when creating a pod?

Answer 1

I found a solution. The problem is actual for Kubernetes version 1.19+ and COS/Ubuntu images based on containerd for GKE nodes. Before the 1.19 version Kubernetes used to use Docker for building images, but now it uses containerd. More details could be found in the official Google Cloud documentation.

You need to create and put an CA certificate to each GKE node. Here you can find an answer how to do it correctly https://stackoverflow.com/a/67724696/3319341.

But for containerd solution you should replace command

nsenter --target 1 --mount systemctl restart docker

by

nsenter --target 1 --mount systemctl restart containerd

A more detailed answer: https://stackoverflow.com/a/67990395/3319341

Answer 2

Install the Root CA certificates on the server. That's it – now the error should be gone. If you don't know the root CA, open the URL that gives you the error in a browser (i.e. Chrome). Click the lock next to the URL and select Certificate (Valid).

Reference link:

https://docs.docker.com/registry/insecure/

https://writeabout.net/2020/03/25/x509-certificate-signed-by-unknown-authority/

Answer 3

Make sure that you have added the certs by moving the root CA cert file into /usr/local/share/ca-certificates and then running sudo update-ca-certificates.

The first step for fixing the issue is to restart the docker so that the system can detect changes in the OS certificates. The docker has an additional location that we can use to trust individual registry server CA. The CA certificate needs to be placed in:

/etc/docker/certs.d/<docker registry>/ca.crt

If we need to include the port number, we need to specify that in the image tag. Eg:

/etc/docker/certs.d/my-registry.example.com:5000/ca.crt

If the above solution does not fix the issue, the following steps needs to be carried out –

X509 errors usually indicate that you are attempting to use a self-signed certificate without configuring the Docker daemon correctly

1: Create a file /etc/docker/daemon.json and add insecure-registries

{
“insecure-registries” : [“docker.domain.com:3000”]
}

Replace “docker.domain.com” with your Docker Registry instance hostname, and the port “3000”, with the port your Docker Registry is running on.

With insecure registries enabled, Docker goes through the following steps:

• First, try using HTTPS.
• If HTTPS is available but the certificate is invalid, ignore the error about the certificate. If HTTPS is not available, fall back to HTTP.

2: Restart the docker daemon by executing the command

systemctl restart docker

3: Create a directory with the same name as the host

mkdir -p /etc/docker/certs.d/docker.domain.com

4: Save the certificate in the newly created directory

ex +’/BEGIN CERTIFICATE/,/END CERTIFICATE/p’ <(echo | OpenSSL s_client -show certs -connect docker.domain.com:443) -suq > /etc/docker/certs.d/docker.domain.com/docker_registry.crt

• If you’re pulling an image from a private registry, make sure that you’ve created a Secret containing the credentials you need to access. Also make sure that you’ve added the Secret in the appropriate namespace.
• You’ll also need to set the imagePullSecrets field on your Pod. This field tells Kubernetes which Secret it should use, when authenticating to the registry.