0

I am attempting to install Kubernetes on VMs running Ubuntu 10.04 LTS, and running into a problem when trying to initialise the system, the kubeadm init command results in failure (full log below).

VM: 2 CPUs, 512mb RAM, 100 gig disk, running under VMWare ESXi6.

OS: Ubuntu 18.04 LTS server install, fully updated via apt update and apt upgrade before beginning the Docker and Kubernetes installs.

Docker installed as per instructions here, install completes with no errors: https://kubernetes.io/docs/setup/production-environment/container-runtimes/#docker

Kubernetes installed as per instructions here, except for the Docker section (as following those instructions produces a PreFlight error re systemd/cgroupfs): https://vitux.com/install-and-deploy-kubernetes-on-ubuntu/

All installation appears to proceed smoothly with no errors reported, however attempting to start Kubernetes then fails, as shown in the log below.

I am entirely new to both Docker and Kubernetes though I get the main concepts and have experimented with the on-line tutorials on kubernetes.io, but until I can get a working system installed I'm unable to progress further. At the point at which kubeadm attempts to start the cluster, everything hangs for the four minutes, and then exits with the timeout as shown below.

root@k8s-master-dev:~# sudo kubeadm init --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.15.3
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master-dev kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.24.0.100]
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master-dev localhost] and IPs [10.24.0.100 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master-dev localhost] and IPs [10.24.0.100 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred:
        timed out waiting for the condition

This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'
        - 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
        - 'docker ps -a | grep kube | grep -v pause'
        Once you have found the failing container, you can inspect its logs with:
        - 'docker logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster

I've had a look at both the log journal data and the docker logs but other than lots of timeouts, can't see anything that explains the actual error. Can anyone advise where I should be looking, and what's most likely to be the cause of the problem?

Things already tried: Removing all IPTables rules and setting defaults to "accept". Running with Docker install as per the vitux.com instructions (gives a PreFlight warning but no errors, but same timeout on attempting to init Kubernetes).

Update: Following from @Crou's comment, here is what happens now if I try just 'kubeadm init' as root:

root@k8s-master-dev:~# uptime
 16:34:49 up  7:23,  3 users,  load average: 10.55, 16.77, 19.31
root@k8s-master-dev:~# kubeadm init
[init] Using Kubernetes version: v1.15.3
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR Port-6443]: Port 6443 is in use
        [ERROR Port-10251]: Port 10251 is in use
        [ERROR Port-10252]: Port 10252 is in use
        [ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
        [ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
        [ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
        [ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
        [ERROR Port-10250]: Port 10250 is in use
        [ERROR Port-2379]: Port 2379 is in use
        [ERROR Port-2380]: Port 2380 is in use
        [ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

Re the very high load shown bu uptime, that starts as soon as the init is first attempted and load remains very high unless a kibeadm reset is done to clear everything down.

  • What is the status of kubelet? Try doing plain kubeadm init as root. – Crou Aug 22 at 14:33
  • @Crou Full output too long for a comment here but it throws lots of fatal errors about ports in use and files already existing. It's as if it is mostly running - but it can't be connected to or operated in any way. – Pyromancer Aug 22 at 16:53
  • You need to check what is causing the load when you do kubectl init after you clear everything by kubectl reset. It looks like it gets stuck somewhere. – Crou Aug 23 at 10:58
0

Further experimentation: In addition to VMware ESXi hypervisors we also run XenServer hypervisors on identical hardware platforms. I attempted an identical install on a VM on one of the Xen blades, however it proved impossible to install Ubuntu at all, install failed at the "Installing the kernel" phase. I tried two separate installs, both failed at exactly the same place. The VM spec was same as under ESXi, 2 cores, 512MB memory, 100 gig hard disk.

Solution: We finally solved the problem by dispensing with VMs and installing Ubuntu 18.04 directly on the hardware, no hypervisors or VMs involved, then adding Docker and Kubernetes, and this time the kubeadm init command completed correctly with the expected messages appearing. The spec of the blade we installed on is 2 x hexacore Xeon processors and 48 gig of ram, with a 1TB hard disk.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.