0

In a nutshell, I munged my kubeadm cluster and am trying to start from scratch (I was not able to do a proper tear down of the cluster, which is why I was forced to go this route of kubeadm reset and sudo yum remove kubeadm kubectl kubelet kubernetes-cni kube* minus the removal of the node from the cluster).

I removed all packages (running on RHEL 7) and am now trying to reinstall, but getting the error on startup:

-- Unit kubelet.service has begun starting up.
Nov 13 12:23:50 thalia2.ahc.umn.edu kubelet[31361]: F1113 12:23:50.681719   31361 server.go:190] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to
read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory
Nov 13 12:23:50 thalia2.ahc.umn.edu systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
Nov 13 12:23:50 thalia2.ahc.umn.edu systemd[1]: Unit kubelet.service entered failed state.
Nov 13 12:23:50 thalia2.ahc.umn.edu systemd[1]: kubelet.service failed.
Nov 13 12:24:00 thalia2.ahc.umn.edu systemd[1]: kubelet.service holdoff time over, scheduling restart.
Nov 13 12:24:00 thalia2.ahc.umn.edu systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kubelet.service has finished starting up.
--

Every post I've seen regarding this error indicates that this file (/var/lib/kubelet/config.yaml) is missing because I did not run kubeadm init, but I am NOT creating a master node - I just want to run this like it was a fresh install as per installing-kubeadm-kubelet-and-kubectl

Am I missing something obvious? I do not remember running kubeadm init on each node before I started the kubelet service in my initial install a couple months back, plus, nowhere in the directions for installing the packages does it say to do a Kubeadm init.

Also of note: On my former master node, I am getting the error after trying a reinstall and start of kubectl:

Get https://134.84.53.162:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dthalia0.ahc.umn.edu&limit=500&resourceVersion=0: dial tcp 134.84.53.162:6443: connect: connection refused
Nov 13 13:40:40 thalia0.ahc.umn.edu kubelet[17689]: W1113 13:40:40.487124   17689 cni.go:188] Unable to update cni config: No networks found in /etc/cni/net.d
Nov 13 13:40:40 thalia0.ahc.umn.edu kubelet[17689]: E1113 13:40:40.487404   17689 kubelet.go:2167] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNot
Ready message:docker: network plugin is not ready: cni config uninitialized

but, I thought I had removed everything by doing a kubeadm reset?

I did delete the calico folder on my former master node (sudo rm -rf /var/etcd/calico-data), and then did a restart of kubectl and am getting this error now:

Nov 13 13:55:37 thalia0.ahc.umn.edu kubelet[29725]: E1113 13:55:37.447381   29725 kubelet.go:2236] node "thalia0.ahc.umn.edu" not found
Nov 13 13:55:37 thalia0.ahc.umn.edu kubelet[29725]: E1113 13:55:37.547732   29725 kubelet.go:2236] node "thalia0.ahc.umn.edu" not found
Nov 13 13:55:37 thalia0.ahc.umn.edu kubelet[29725]: E1113 13:55:37.648002   29725 kubelet.go:2236] node "thalia0.ahc.umn.edu" not found
Nov 13 13:55:37 thalia0.ahc.umn.edu kubelet[29725]: E1113 13:55:37.748219   29725 kubelet.go:2236] node "thalia0.ahc.umn.edu" not found
Nov 13 13:55:37 thalia0.ahc.umn.edu kubelet[29725]: E1113 13:55:37.849201   29725 kubelet.go:2236] node "thalia0.ahc.umn.edu" not found
Nov 13 13:55:37 thalia0.ahc.umn.edu kubelet[29725]: E1113 13:55:37.950018   29725 kubelet.go:2236] node "thalia0.ahc.umn.edu" not found
Nov 13 13:55:38 thalia0.ahc.umn.edu kubelet[29725]: E1113 13:55:38.050193   29725 kubelet.go:2236] node "thalia0.ahc.umn.edu" not found
Nov 13 13:55:38 thalia0.ahc.umn.edu kubelet[29725]: E1113 13:55:38.150479   29725 kubelet.go:2236] node "thalia0.ahc.umn.edu" not found
Nov 13 13:55:38 thalia0.ahc.umn.edu kubelet[29725]: I1113 13:55:38.173736   29725 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
Nov 13 13:55:38 thalia0.ahc.umn.edu kubelet[29725]: I1113 13:55:38.180154   29725 kubelet_node_status.go:70] Attempting to register node thalia0.ahc.umn.edu

So, it would appear that there is "residue" from my old cluster. Is there some hidden package out there that a kubeadm reset and a sudo yum remove kubeadm kubectl kubelet kubernetes-cni kube* did not delete?

  • Going to have our server ops people do a backup restore on all nodes. Hopefully this will resolve the initial issue that drove me to this madness in the first place. – horcle_buzz Nov 13 '18 at 23:00
0

I thought I had removed everything by doing a kubeadm reset?

regarding to documentation it is not delete everything. it just reverse the changes, so where you delete the packages and just broke all paths, run kubeadm init to reinitialise cluster.

0

The issue ended up being that the /var directory had gotten full. Cleaned it out and now everything starts as expected.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.