Kubernetes, how do I let kubernetes use the internal network for node to node communication?

Question

Context.

I am following the basic kubernetes installation here (in Hetzner Cloud, if it is needed). 1 controller and 1 worker.

All is fine apparently.

• the servers have an external IP interface (public Ipv4) and one internal one (normally 10.0.0.2 or 10.0.0.3)
• The controller goes up
• I install flannel
• I set options with kubeadm init and I change the kubectl config to use only the internal ip (otherwise it goes showing the external ip as "internal ip" of the nodes). Namely

private_ipaddr=$( ifconfig eth1 | grep -i inet | head -1 | awk '{print $2}' ) #eth0 is the public Ipv4
echo "KUBELET_EXTRA_ARGS='--node-ip ${private_ipaddr}'" > /etc/sysconfig/kubelet
systemctl daemon-reload
systemctl restart kubelet

kubeadm init --apiserver-advertise-address=$private_ipaddr --pod-network-cidr=10.244.0.0/16

• then I join the worker, and the advertised address for the api is indeed internal (10.0.0.3 for example)
• kubectl get nodes -o wide shows two nodes ready after few minutes.
• then I start to deploy one pod. The pod is a simply busybox or alpine.
• the pod gets deployed on the only worker.
• I try to reach from the deployed pod the fqdn of the coredns pod. It does not work.
• There is no firewall set on the system aside from kubernetes chains. Between the systems there is no firewall if using the internal network, while the traffic on the public ipv4 is firewalled (except port 22)
• What I discovered is: if I open the firewall between the two public ipv4 of the servers, then the connection between pod and core-dns pod works. Otherwise it doesn't .

This leds me to realize that despite the node-ip is set to the internal ip, for the communications kubernetes still goes over the public ipv4 interface, rather than communicating via internal network.

Hence the question. How can I say to kubernetes to use only the internal network and not the external one?

Answer 1

There is one idea why it can happen this way. Based on the flannel configuration doc, there is an option:

--iface="": interface to use (IP or name) for inter-host communication. Defaults to the interface for the default route on the machine. This can be specified multiple times to check each option in order. Returns the first match found.

In order to pass this option, you'll need to download the flannel manifest in yaml and add this part to args in daemonSet - containers - kube-flannel one, so it should look like:

  containers:
  - name: kube-flannel
    image: quay.io/coreos/flannel:v0.14.0
    command:
    - /opt/bin/flanneld
    args:
    - --ip-masq
    - --kube-subnet-mgr
    - --iface=interface_name

And then kubectl apply -f kube-flannel.yaml

It may require to delete flannel and apply from scratch.

Upd:

You can check which interface is used in kube-flannel pod logs:

kubectl logs -n kube-system kube-flannel-ds-xxxxx -c kube-flannel | grep interface

In my case it was one and only network interface:

I0611 12:21:47.303175       1 main.go:520] Determining IP address of default interface
I0611 12:21:47.303636       1 main.go:533] Using interface with name ens4 and address 10.186.0.2
I0611 12:21:47.303668       1 main.go:550] Defaulting external address to interface address (10.186.0.2)