kubectl get nodes error: You must be logged in to the server (Unauthorized) - how to fix

Question

Without changes in infrastructure If I execute any kubectl command, ie:

kubectl get nodes

I got error

You must be logged in to the server (Unauthorized) 

And I had working kubernetes cluster and did no changes to it... Any ideas how to debug this? kubectl has no -vv od debug flag to give more information.

If i try

kubectl version

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0",
(...) 
error: You must be logged in to the server (the server has asked for the client to provide credentials)

Answer 1

In my case the issue started after renewing kubernates certificates, this caused the existing ~/.kube/config to have outdated keys and certificate values in it.

The solution was to replace the values client-certificate-data and client-key-data in file ~/.kube/config with the values from the updated file in /etc/kubernetes/kubelet.conf of the same name.

Answer 2

You can copy the client-certificate-data and client-key-data from /etc/kubernetes/admin.conf to your ~/.kube/config file as of more recent versions of Kubernetes. See this answer for determining when your certificates expire.

Answer 3

I have faced the similar issue today and the above comments helped me to fix the issue. I am adding more details with my scenario because it might be helpful for the people which have similar settings.

I have a separate user for connecting to my k8s cluster (It's a normal cluster in EC2 instances). I had created the user arunlal with limited access by adding ClusterRoleBindings.

If you get the following error while running API to cluster (in my case Kubectl):

error: You must be logged in to the server (Unauthorized)

Go through the following order.

- First check the cert used in your config file (local client)

I had a separate configuration on my local system, because the main config had the details about the other k8s & okd cluster credentials. So I had created second configuration on my Laptop (/Users/arunlal/.kube/config_two). In this case I have the following aliases:

alias kctl="kubectl --kubeconfig=/Users/arunlal/.kube/config_two"

- From this file you will get the cert that we are using.

[arunlal.as@crybit.com ~] cat /Users/arunlal/.kube/config_two| grep -A 5 users
users:
- name: arunlal
  user:
    client-certificate: /Users/arunlal/.arunlal-keys/arunlal.crt
    client-key: /Users/arunlal/.arunlal-keys/arunlal.key

- Once you get the cert in your client configuration you can check the validity using the openssl command.

    [arunlal.as@crybit.com ~] openssl x509 -noout -dates -in /Users/arunlal/.arunlal-keys/arunlal.crt
    notBefore=Jun 22 23:43:22 2021 GMT
    notAfter=Sep 30 23:43:22 2021 GMT

- Validate the expiry

While creating the user I passed the days as 5, that was the issue. How I created user?

openssl genrsa -out arunlal.key 2048
openssl req -new -key arunlal.key -out arunlal.csr -subj "/CN=arunlal/O=crybit"
openssl x509 -req -in arunlal.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out arunlal.crt -days 5
kubectl config set-credentials arunlal --client-certificate=/root/arunlal-keys/arunlal.crt  --client-key=/root/arunlal-keys/arunlal.key

- To fix, I recreated the cert with more number of days

openssl x509 -req -in arunlal.csr -CA /etc/kubernetes/pki/ca.crt
-CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out arunlal.crt -days 100

- This we need to run from the k8s cluster.

- Replaced the cert locally.

Modified /Users/arunlal/.arunlal-keys/arunlal.crt with new cert.

Hope this will help someone. Thanks!

~ arun

Answer 4

I got the same after updating certificates:

kubeadm alpha certs renew all

And then I had to follow

$ cd ~/.kube

# Archive the old config file containing the out of date certificates
$ mv config conf.archive.2021

# Copy the new configuration file created using kubeadm
$ cp /etc/kubernetes/admin.conf config

# apply permissions to your current admin user and group
$ sudo chown $(id -u):$(id -g) config

Kubernetes version : 1.19

Reference

Answer 5

The kubeconfig certificate may have changed. If you deployed your cluster using terraform. Do terraform apply to generate a new kubeconfig file.

Answer 6

In my case I was using a cluster created by kOps and cluster admin user with credentials (~/.kube/config) generated by: kops export kubeconfig --admin

By default the credentials expire after 18 hours. So another:

kops export kubeconfig --admin

Did it for me.

Answer 7

Fixed - after all the cert has changed. Check your ~/.kube/config if you have this

Answer 8

I'm getting "error: You must be logged in to the server (the server has asked for the client to provide credentials)" in Azure Kubernetes service.

I fixed this with trying to connect and executed the command mentioned on the Connect to ClusterName.

az account set --subscription de81a6e3-1784-4732-9282-XXXXXXXX7
az aks get-credentials --resource-group resourceGroupName --name clusterName

Answer 9

I fixed the issue by deleting minikube

minikube delete 
minikube start --vm-driver=none

Answer 10

After renewing your certs you need to apply the new certs to the admin config for kubectl to work.

The following takes a backup of your existing config, and applies the new admin config.

cp /root/.kube/config /root/.kube/.old-$(date --iso)-config
cp /etc/kubernetes/admin.conf /root/.kube/config

Answer 11

In my case using AWS EKS what solved the problem was:

aws eks --region <region code> update-kubeconfig --name <cluster name>