I have a kubeadm installed kubernetes cluster. Recently it stopped working. kubelet is running but seems stuck in initialization phases. I think the root cause is this recurring log in kube-apiserver:

1 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate has expired or is not yet valid: current time 2021-06-02T13:18:50Z is after 2021-05-29T15:48:22Z

So there is a certificate issue, also kubectl is failing with unauthorized. The thing is, kubeadm certs check-expiration seems happy, and I even manually checked a few yaml config files (base64 decoded certificates, and run them through openssl to check the date). Nevertheless, I asked kubeadm to renew all certificates and rebooted everything, to no effect.

Any idea how I can identify which certificate exactly has expired ?


[acknowledgment and reference] I was helped by a kubernetes' dev here

The expired certificate was /var/lib/kubelet/pki/kubelet/pki/kubelet-client-2020-*.pem. The certificates in /var/lib/kublet/pki/ are not handled by kubeadm cert but by kubelet itself, so it's supposed to be renewed automatically, but for some reason this didn't happen as planned for us. The kubelet-client-current.pem had been renewed, but something was still using an old (and expired) certificate.

Here is how I fixed the issue:

  • /etc/kubernetes/kubelet.conf was obsolete, in particular using default-user instead of system:node:node_name. I deleted the file, created a kubeadm conf file and ran kubeadm init phase kubeconfig kubelet to recreate a clean kubelet.conf
  • /var/lib/kublet/pki/kubelet-client-current.pem is supposed to be a symlink, which was not the case for me. So I removed it.
  • restart kubelet and apiserver (kill the pod using containerd, docker, etc. since kubectl is unavailable) and wait for a new kubelet-client-current.pem to be created ; it should be a symlink.
  • run kubeadm init phase kubelet-finalize all
  • restart kubelet again
  • run kubeadm certs renew all
  • reboot (or restart kubelet and all control plane pods)
  • update your kubectl conf from /etc/kubernetes/admin.conf

Similar issues seems to be linked to NTP desynchronization.
Try forcing time synchronization (run as root):

# service ntp stop
# ntpd -gq
# service ntp start
  • Thanks for the suggestion. Time was correct, and I resynchronized with no effect.
    – Antoine
    Jun 3 '21 at 12:38

Renew Kubernetes certificates (RUN on all master node)

#kubeadm alpha certs check-expiration
#kubeadm init phase kubelet-finalize all
#kubeadm alpha certs renew all
#cd /etc/kubernetes
#kubeadm alpha kubeconfig user --org system:nodes --client-name system:node:$(hostname) > kubelet.conf
#mkdir -p $HOME/.kube
#cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
#chown $(id -u):$(id -g) $HOME/.kube/config
#restart server
#kubectl get nodes

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.