0

(This questions has been moved from Stackoverflow)

First I apologize for the lengthy entry but I think its better to give as much detail as possible.

  • Host OS: Win10
  • Guest OS: Ubuntu 20.10 (Groovy)
  • Docker CE: 5:19.03.15~3-0~ubuntu-bionic
  • Kubernetes: 1.20.4-00
  • VirtualBox: 6.1.18 on Win10
    • eth0: NAT
    • eth1: Host only (192.168.50.1/24)

I have three control-plane nodes with a keepalived/haproxy combination installed on each of them as a "load balancer" with an IP of 192.168.50.100. As a consequence the apiserver entrypoint is 'poc-lb:8443' which in turn is distributed among the control-plane nodes on port 6443. /etc/hosts on each of the nodes looks like:

  • 192.168.50.10 poc-ctrl-1
  • 192.168.50.11 poc-ctrl-2
  • 192.168.50.12 poc-ctrl-3
  • 192.168.50.100 poc-lb

I initialize the k8s cluster on poc-ctrl-1 using:

sudo kubeadm init --apiserver-advertise-address 192.168.50.10 --control-plane-endpoint poc-lb:8443 --upload-certs

When it has been initialized on that node I deploy the weave CNI plugin using:

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

After the weave plugin has been deployed on the first control-plane node, I join the second and third control-plane nodes (poc-ctrl-2 & poc-ctrl-3) using a 'kubeadm join' command (--token, discovery-token and --certificate-key have been removed for brevity):

sudo kubeadm join poc-lb:8443 --control-plane --apiserver-advertise-address 192.168.50.11
sudo kubeadm join poc-lb:8443 --control-plane --apiserver-advertise-address 192.168.50.12

The nodes join without a problem, however, the weave PODs don't seem to be very happy. This is the log for the 'weave' container on poc-ctrl-1:

DEBU: 2021/03/08 15:03:32.486479 [kube-peers] Checking peer "1e:85:5b:9b:50:c5" against list &{[]}
Peer not in list; removing persisted data
INFO: 2021/03/08 15:03:32.561859 Command line options: map[conn-limit:200 datapath:datapath db-prefix:/weavedb/weave-net docker-api: expect-npc:true http-addr:127.0.0.1:6784 ipalloc-init:consensus=0 ipalloc-range:10.32.0.0/12 metrics-addr:0.0.0.0:6782 name:1e:85:5b:9b:50:c5 nickname:poc-ctrl-1 no-dns:true no-masq-local:true port:6783]
INFO: 2021/03/08 15:03:32.561901 weave  2.8.1
INFO: 2021/03/08 15:03:33.216812 Bridge type is bridged_fastdp
INFO: 2021/03/08 15:03:33.216846 Communication between peers is unencrypted.
INFO: 2021/03/08 15:03:33.224064 Our name is 1e:85:5b:9b:50:c5(poc-ctrl-1)
INFO: 2021/03/08 15:03:33.224115 Launch detected - using supplied peer list: []
INFO: 2021/03/08 15:03:33.224149 Using "no-masq-local" LocalRangeTracker
INFO: 2021/03/08 15:03:33.224155 Checking for pre-existing addresses on weave bridge
INFO: 2021/03/08 15:03:33.233984 [allocator 1e:85:5b:9b:50:c5] No valid persisted data
INFO: 2021/03/08 15:03:33.262924 [allocator 1e:85:5b:9b:50:c5] Initialising via deferred consensus
INFO: 2021/03/08 15:03:33.263027 Sniffing traffic on datapath (via ODP)
INFO: 2021/03/08 15:03:33.265856 Listening for HTTP control messages on 127.0.0.1:6784
INFO: 2021/03/08 15:03:33.266928 Listening for metrics requests on 0.0.0.0:6782
INFO: 2021/03/08 15:03:33.401417 Error checking version: Get "https://checkpoint-api.weave.works/v1/check/weave-net?arch=amd64&flag_docker-version=none&flag_kernel-version=5.8.0-41-generic&os=linux&signature=aQyw2dVd0f8HNRaTeZ8N3lnlww9j0P3J5P359AkeBBk%3D&version=2.8.1": dial tcp: lookup checkpoint-api.weave.works on 10.96.0.10:53: write udp 10.0.2.15:46287->10.96.0.10:53: write: operation not permitted
INFO: 2021/03/08 15:03:33.578810 [kube-peers] Added myself to peer list &{[{1e:85:5b:9b:50:c5 poc-ctrl-1}]}
DEBU: 2021/03/08 15:03:33.588343 [kube-peers] Nodes that have disappeared: map[]
INFO: 2021/03/08 15:03:33.599543 Assuming quorum size of 1
INFO: 2021/03/08 15:03:33.599784 adding entry 10.32.0.0/12 to weaver-no-masq-local of 0
INFO: 2021/03/08 15:03:33.599809 added entry 10.32.0.0/12 to weaver-no-masq-local of 0
10.32.0.1
DEBU: 2021/03/08 15:03:33.684752 registering for updates for node delete events
INFO: 2021/03/08 15:20:34.605758 ->[192.168.50.12:57361] connection accepted
INFO: 2021/03/08 15:20:34.620605 ->[192.168.50.12:57361|a2:18:ea:75:33:ca(poc-ctrl-3)]: connection ready; using protocol version 2
INFO: 2021/03/08 15:20:34.620811 overlay_switch ->[a2:18:ea:75:33:ca(poc-ctrl-3)] using fastdp
INFO: 2021/03/08 15:20:34.620830 ->[192.168.50.12:57361|a2:18:ea:75:33:ca(poc-ctrl-3)]: connection added (new peer)
INFO: 2021/03/08 15:20:34.634204 ->[192.168.50.12:57361|a2:18:ea:75:33:ca(poc-ctrl-3)]: connection fully established
INFO: 2021/03/08 15:20:34.723969 sleeve ->[192.168.50.12:6783|a2:18:ea:75:33:ca(poc-ctrl-3)]: Effective MTU verified at 1438
INFO: 2021/03/08 15:20:35.742452 Discovered remote MAC a2:18:ea:75:33:ca at a2:18:ea:75:33:ca(poc-ctrl-3)
INFO: 2021/03/08 15:20:36.352445 Discovered remote MAC ee:27:39:76:a7:5d at a2:18:ea:75:33:ca(poc-ctrl-3)
INFO: 2021/03/08 15:20:36.510082 Discovered remote MAC be:c8:b2:c2:d2:cf at a2:18:ea:75:33:ca(poc-ctrl-3)
INFO: 2021/03/08 15:21:04.875787 adding entry 10.32.0.0/13 to weaver-no-masq-local of 0
INFO: 2021/03/08 15:21:04.875840 added entry 10.32.0.0/13 to weaver-no-masq-local of 0
INFO: 2021/03/08 15:21:04.876883 adding entry 10.40.0.0/14 to weaver-no-masq-local of 0
INFO: 2021/03/08 15:21:04.876905 added entry 10.40.0.0/14 to weaver-no-masq-local of 0
INFO: 2021/03/08 15:21:04.877778 deleting entry 10.32.0.0/12 from weaver-no-masq-local of 0
INFO: 2021/03/08 15:21:04.877792 deleted entry 10.32.0.0/12 from weaver-no-masq-local of 0

This is the log for the 'weave' container on poc-ctrl-2:

DEBU: 2021/03/08 15:40:06.625988 [kube-peers] Checking peer "9a:7c:0f:a1:76:36" against list &{[{1e:85:5b:9b:50:c5 poc-ctrl-1}]}
Peer not in list; removing persisted data
FATA: 2021/03/08 15:40:36.654217 [kube-peers] Could not get Kubernetes version: Get "https://10.96.0.1:443/version?timeout=32s": dial tcp 10.96.0.1:443: i/o timeout

And, finally, the log for the 'weave' container on poc-ctrl-3:

FATA: 2021/03/08 15:21:04.964921 [kube-peers] Could not update peer list: Unable to fetch ConfigMap kube-system/weave-net: Get "https://10.96.0.1:443/api/v1/namespaces/kube-system/configmaps/weave-net": dial tcp 10.96.0.1:443: i/o timeout
INFO: 2021/03/08 15:21:04.981699 adding entry 10.44.0.0/14 to weaver-no-masq-local of 0
INFO: 2021/03/08 15:21:04.981948 added entry 10.44.0.0/14 to weaver-no-masq-local of 0
10.44.0.0
INFO: 2021/03/08 15:21:16.935459 ->[192.168.50.11:6783] attempting connection
INFO: 2021/03/08 15:21:16.936059 ->[192.168.50.11:6783] error during connection attempt: dial tcp :0->192.168.50.11:6783: connect: connection refused
FATA: 2021/03/08 15:21:35.037984 [kube-peers] could not set node status: Patch "https://10.96.0.1:443/api/v1/nodes/poc-ctrl-3/status": dial tcp 10.96.0.1:443: i/o timeout
INFO: 2021/03/08 15:21:40.255913 ->[192.168.50.11:6783] attempting connection
INFO: 2021/03/08 15:21:40.256478 ->[192.168.50.11:6783] error during connection attempt: dial tcp :0->192.168.50.11:6783: connect: connection refused
INFO: 2021/03/08 15:21:59.917279 Discovered remote MAC 4a:0d:3e:de:62:b4 at 1e:85:5b:9b:50:c5(poc-ctrl-1)
INFO: 2021/03/08 15:22:30.157989 ->[192.168.50.11:6783] attempting connection
INFO: 2021/03/08 15:22:30.158579 ->[192.168.50.11:6783] error during connection attempt: dial tcp :0->192.168.50.11:6783: connect: connection refused
INFO: 2021/03/08 15:23:25.508244 ->[192.168.50.11:6783] attempting connection
INFO: 2021/03/08 15:23:25.508785 ->[192.168.50.11:6783] error during connection attempt: dial tcp :0->192.168.50.11:6783: connect: connection refused
INFO: 2021/03/08 15:24:57.982083 ->[192.168.50.11:6783] attempting connection
INFO: 2021/03/08 15:24:57.982653 ->[192.168.50.11:6783] error during connection attempt: dial tcp :0->192.168.50.11:6783: connect: connection refused
INFO: 2021/03/08 15:26:10.300785 ->[192.168.50.11:6783] attempting connection
INFO: 2021/03/08 15:26:10.301685 ->[192.168.50.11:6783] error during connection attempt: dial tcp :0->192.168.50.11:6783: connect: connection refused
INFO: 2021/03/08 15:27:42.395131 ->[192.168.50.11:6783] attempting connection
INFO: 2021/03/08 15:27:42.395556 ->[192.168.50.11:6783] error during connection attempt: dial tcp :0->192.168.50.11:6783: connect: connection refused
INFO: 2021/03/08 15:34:00.374000 ->[192.168.50.11:6783] attempting connection
INFO: 2021/03/08 15:34:00.374547 ->[192.168.50.11:6783] error during connection attempt: dial tcp :0->192.168.50.11:6783: connect: connection refused
INFO: 2021/03/08 15:40:56.090626 ->[192.168.50.11:6783] attempting connection
INFO: 2021/03/08 15:40:56.091130 ->[192.168.50.11:6783] error during connection attempt: dial tcp :0->192.168.50.11:6783: connect: connection refused

All of the nodes have the 'br_netfilter' loaded and net.bridge.bridge-nf-call-iptables = 1.

The IP 10.96.0.1 is assigned to the kubernetes service on 443/tcp and ports 6783/tcp and 678(3|4)/udp are used by weave. Given the outputs above I get the feeling I have some iptables related issues and/or could it be that the packets travels the default route on the (eth0 interface)?

ip route gives:

default via 10.0.2.2 dev eth0 proto dhcp src 10.0.2.15 metric 100
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15
10.0.2.2 dev eth0 proto dhcp scope link src 10.0.2.15 metric 100
10.32.0.0/12 dev weave proto kernel scope link src 10.32.0.1
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.50.0/24 dev eth1 proto kernel scope link src 192.168.50.10

What have I missed here?

1

After inspecting the iptables rules I got a feeling that the IP assigned to the k8s svc MUST be routed to the "wrong" interface. I issued

sudo ip route add 10.96.0.1 dev eth1

and weave started!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.