kubernetes flannel daemonset not starting (Clean Ubuntu 16 and 18)

Question

I just installed kubernetes on a clean ubuntu machine following this steps (from linux academy):

As root:

apt install -y docker.io

cat << EOF >/etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

curl -s http://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -

cat << EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF

apt update -y

apt install -y kubeadm kubectl kubelet

Then as regular user initialize kubernetes:

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

At this moment the node is set but not ready yet so I run yml for flannel to create the daemon, services and pods:

sudo kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml

But, I can't see the pods and the daemonsets got desired pods = 0

No pods (coredns are waiting for flannel setup):

NAMESPACE     NAME                                                 READY   STATUS    RESTARTS   AGE
kube-system   coredns-576cbf47c7-j2rbn                             0/1     Pending   0          2m25s
kube-system   coredns-576cbf47c7-lqhrj                             0/1     Pending   0          2m25s
kube-system   etcd-webdriver1.mylabserver.com                      1/1     Running   0          118s
kube-system   kube-apiserver-webdriver1.mylabserver.com            1/1     Running   0          94s
kube-system   kube-controller-manager-webdriver1.mylabserver.com   1/1     Running   0          89s
kube-system   kube-proxy-fzh97                                     1/1     Running   0          2m25s
kube-system   kube-scheduler-webdriver1.mylabserver.com            1/1     Running   0          90s

And the deamonset

NAMESPACE     NAME                             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                   AGE
kube-system   daemonset.apps/kube-flannel-ds   0         0         0       0            0           beta.kubernetes.io/arch=amd64   3m49s
kube-system   daemonset.apps/kube-proxy        1         1         1       1            1           <none>                          48m

Any help how can I debug this issue?

Answer 1

The configuration file you're applying to create the flannel resources includes daemonset tolerations that are too restrictive, so the pods won't be scheduled on any nodes. Add a catchall scheduling toleration to the flannel daemonset (like the canal configuration does) and they'll be scheduled as expected. You can do this in 2 ways:

(1) Patch your existing configuration

kubectl patch daemonset kube-flannel-ds \
  --namespace=kube-system \
  --patch='{"spec":{"template":{"spec":{"tolerations":[{"key": "node-role.kubernetes.io/master", "operator": "Exists", "effect": "NoSchedule"},{"effect":"NoSchedule","operator":"Exists"}]}}}}'

(2) Modify the configuration file to include the following before applying it:

...
spec:
  template:
    ...
    spec:
      ...
      tolerations:
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
        - operator: Exists
          effect: NoSchedule
      ...

This was an issue for my clusters when upgrading from k8s v1.10. Looks to be related to changing taints/tolerations in different k8s versions.