0

I have a machine running as a Kubernetes master. It's also a node in its cluster. I want to stop all the masterish stuff and leave it as a node I can join elsewhere. I also have the following constraints on it:

  1. No internet access (so I can't load software from repos onto it).
  2. The "master" is currently configured for a subnet it is not on so I can't actually talk to it (as is) with kubectl.

I'm hoping I can just systemctl stop some services (e.g., etcd, kube-{apiserver,controller-manager,scheduler} and/or some docker containers and that's it. (Plus then kubeadm reset and run the new join script of course.) But at this stage I don't even know whether systemctl stop on the services keeps the docker containers from running, or vice versa.

(Of course if more is necessary in order to fully 'reset' this to be a plain node than just kubeadm reset - like deleting configuration files - please explain that too.)

[Kubernetes v1.11.10, Ubuntu 16.04]

(If, to answer the question, you need to know how I got wedged this way I can edit this answer to add that but at this time it seems to me to be not needed to answer and it's tedious.)

(Also please don't suggest starting over including getting the internet connection back - e.g. to load software - because that can't happen.)

3
  • Do I get it right? You want to remove master(decommission the cluster) and attache it to a different cluster as a worker node?
    – Crou
    Aug 20, 2019 at 14:25
  • @Crou - yes to both thanks!
    – davidbak
    Aug 20, 2019 at 15:38
  • I'll try to test that and get to you back tomorrow
    – Crou
    Aug 20, 2019 at 15:40

1 Answer 1

2

I've tested this scenario on Kubernetes v1.15.3.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

I've done kubeadm reset on master node.

$ kubeadm reset
[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
[reset] Removing info for node "instance-1" from the ConfigMap "kubeadm-config" in the "kube-system" Namespace
W0821 11:34:26.923767   22594 removeetcdmember.go:61] [reset] failed to remove etcd member: error syncing endpoints with etc: etcdclient: no available endpoints
.Please manually remove this etcd member using etcdctl
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/etcd /var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually.
For example:
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.

As recommended I've flushed iptables and removed $HOME/.kube/ directory.

After that I've joined the server to another cluster as worker node using kubeadm join:

$ kubeadm join 10.128.15.232:6443 --token <CUT> --discovery-token-ca-cert-hash sha256:<CUT>
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
        [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.1. Latest validated version: 18.09
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

This seems to work just fine and as explained in the docs kubeadm reset.

Run this to revert any changes made to this host by ‘kubeadm init’ or ‘kubeadm join’

The “reset” command executes the following phases:

preflight              Run reset pre-flight checks
update-cluster-status  Remove this node from the ClusterStatus object.
remove-etcd-member     Remove a local etcd member.
cleanup-node           Run cleanup node.
$ kubectl get nodes
NAME         STATUS   ROLES    AGE     VERSION
instance-1   Ready    <none>   5m36s   v1.15.3
node2        Ready    master   18m     v1.15.3

If you upgraded your k8s to v1.15.0 you could use kubeadm reset phase.

command with the aim of making kubeadm more modular. This modularity enables you to invoke atomic sub-steps of the reset process. Hence, you can let kubeadm do some parts and fill in yourself where you need customizations.

kubeadm reset phase is consistent with the kubeadm reset workflow, and behind the scene both use the same code.

1
  • Much thanks for going to this effort. Let me try it out!
    – davidbak
    Aug 21, 2019 at 13:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .