2

I have a kubeadm installed kubernetes cluster. Recently it stopped working. kubelet is running but seems stuck in initialization phases. I think the root cause is this recurring log in kube-apiserver:

1 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate has expired or is not yet valid: current time 2021-06-02T13:18:50Z is after 2021-05-29T15:48:22Z

So there is a certificate issue, also kubectl is failing with unauthorized. The thing is, kubeadm certs check-expiration seems happy, and I even manually checked a few yaml config files (base64 decoded certificates, and run them through openssl to check the date). Nevertheless, I asked kubeadm to renew all certificates and rebooted everything, to no effect.

Any idea how I can identify which certificate exactly has expired ?

Improve this question

4 Answers 4

Reset to default
2

[acknowledgment and reference] I was helped by a kubernetes' dev here

The expired certificate was /var/lib/kubelet/pki/kubelet/pki/kubelet-client-2020-*.pem. The certificates in /var/lib/kublet/pki/ are not handled by kubeadm cert but by kubelet itself, so it's supposed to be renewed automatically, but for some reason this didn't happen as planned for us. The kubelet-client-current.pem had been renewed, but something was still using an old (and expired) certificate.

Here is how I fixed the issue:

  • /etc/kubernetes/kubelet.conf was obsolete, in particular using default-user instead of system:node:node_name. I deleted the file, created a kubeadm conf file and ran kubeadm init phase kubeconfig kubelet to recreate a clean kubelet.conf
  • /var/lib/kublet/pki/kubelet-client-current.pem is supposed to be a symlink, which was not the case for me. So I removed it.
  • restart kubelet and apiserver (kill the pod using containerd, docker, etc. since kubectl is unavailable) and wait for a new kubelet-client-current.pem to be created ; it should be a symlink.
  • run kubeadm init phase kubelet-finalize all
  • restart kubelet again
  • run kubeadm certs renew all
  • reboot (or restart kubelet and all control plane pods)
  • update your kubectl conf from /etc/kubernetes/admin.conf
Improve this answer
1
2

If you are a user and have a kubeconfig file and wanna inspect the kubeconfig cert's expiration date you can run this command

cat <path-to-kubeconfig-file> | grep client-certificate-data | cut -f2 -d : | tr -d ' ' | base64 -d | openssl x509 -text -out - | grep "Not After"

Replace your file path with <path-to-kubeconfig-file>

If you wanna find out more details about your cert you can print all details with:

cat <path-to-kubeconfig-file> | grep client-certificate-data | cut -f2 -d : | tr -d ' ' | base64 -d | openssl x509 -text -out -
Improve this answer
1
  • My config has certificate-authority-data instead of client-certificate-data.
    – mpen
    Jan 7 at 20:48
0

Similar issues seems to be linked to NTP desynchronization.
Try forcing time synchronization (run as root):

# service ntp stop
# ntpd -gq
# service ntp start
Improve this answer
1
  • Thanks for the suggestion. Time was correct, and I resynchronized with no effect.
    – Antoine
    Jun 3, 2021 at 12:38
0

Renew Kubernetes certificates (RUN on all master node)

#kubeadm alpha certs check-expiration
#kubeadm init phase kubelet-finalize all
#kubeadm alpha certs renew all
#cd /etc/kubernetes
#kubeadm alpha kubeconfig user --org system:nodes --client-name system:node:$(hostname) > kubelet.conf
#mkdir -p $HOME/.kube
#cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
#chown $(id -u):$(id -g) $HOME/.kube/config
#restart server
#kubectl get nodes
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .