Kubernetes/Flannel doens't work in private network

Question

I am able to run kubernetes (via kubeadm) on a private cluster/cloud without issue only when the nodes in the cluster have access to the internet (all node are connected to a dumb switch, which then has a network cable out to a gateway).

If I remove the network cable that gives nodes access to the internet and then try to setup my cluster and containers again. I get the following error on flannel:

I0720 21:55:48.351628       1 main.go:459] Using interface with name eth2 and address 192.168.10.1
I0720 21:55:48.351782       1 main.go:476] Defaulting external address to interface address (192.168.10.1)
E0720 21:55:48.358553       1 main.go:223] Failed to create SubnetManager: error retrieving pod spec for 'kube-system/kube-flannel-ds-zr17s': Get https://10.96.0.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-ds-zr17s: dial tcp 10.96.0.1:443: connect: network is unreachable

I'm having a hard time understand why would flannel need accesses to the internet? FYI, when connect via switch, I am able to communicate between nodes without issue using static IPs

Answer 1

I was having the same problem. I believe the issue is that when the Internet connection goes away, the default route disappears and flannel can no longer bootstrap itself on that node. Just make sure that your nodes have a default route configured.

You can check by running:

$ ip route

If no default route is listed you can add one from the command-line like this:

$ ip route add default via <gateway_ip> dev <net_device>

where <gateway_ip> is the IP address of your "gateway" and <net_device> is eth0 (or whatever network device name is relevant in your case).