I have successfully deployed a Rancher server on one out of five servers obtained from Hetzner. For the cluster configuration, I designated one of the remaining servers for the control plane and etcd roles, and the other three servers were set up as worker nodes. This configuration resulted in a functional cluster running Kubernetes version v1.27.11+k3s1. However, I encounter an issue when trying to add worker nodes from providers other than Hetzner. Specifically, the process gets stuck at the "Waiting for agent to check in and apply initial plan" phase.
To diagnose potential connectivity problems, I performed network connectivity tests using telnet and dig from the server I was attempting to add, but did not identify any obvious issues. The command used to add worker nodes was as follows:
curl -fL https://rancher.mydomain.org/system-agent-install.sh | sudo sh -s - --server https://rancher.mydomain.org --label 'cattle.io/os=linux' --token ******* --worker
Checking the system logs with sudo journalctl -u k3s-agent --no-pager, I noticed several attempts by the system to perform actions, but encountering errors related to obtaining CA certificates:
root@ubuntu:~# sudo journalctl -u k3s-agent --no-pager
Mar 16 12:12:48 ubuntu systemd[1]: k3s-agent.service: Succeeded.
Mar 16 12:12:48 ubuntu systemd[1]: Stopped Lightweight Kubernetes.
Mar 16 12:12:48 ubuntu systemd[1]: Starting Lightweight Kubernetes...
Mar 16 12:12:48 ubuntu sh[885689]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Mar 16 12:12:48 ubuntu k3s[885707]: time="2024-03-16T12:12:48Z" level=info msg="Starting k3s agent v1.27.11+k3s1 (06d6bc80)"
Mar 16 12:12:48 ubuntu k3s[885707]: time="2024-03-16T12:12:48Z" level=info msg="Adding server to load balancer k3s-agent-load-balancer: [2a01:4f8:c012:3140::1]:6443"
Mar 16 12:12:48 ubuntu k3s[885707]: time="2024-03-16T12:12:48Z" level=info msg="Running load balancer k3s-agent-load-balancer 127.0.0.1:6444 -> [[2a01:4f8:c012:3140::1]:6443] [default: [2a01:4f8:c012:3140::1]:6443]"
Mar 16 12:12:48 ubuntu k3s[885707]: time="2024-03-16T12:12:48Z" level=error msg="failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:45838->127.0.0.1:6444: read: connection reset by peer"
Conversely, when adding servers from Hetzner using the same method, the process completes successfully without any similar issues. For comparison, here's an excerpt from the logs of a Hetzner server being added successfully:
root@dev-k3s-worker-3:~# sudo journalctl -u k3s-agent --no-pager
Mar 15 14:57:53 dev-k3s-worker-3 systemd[1]: Starting Lightweight Kubernetes...
Mar 15 14:57:53 dev-k3s-worker-3 sh[1391]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Mar 15 14:57:53 dev-k3s-worker-3 k3s[1396]: time="2024-03-15T14:57:53Z" level=info msg="Acquiring lock file /var/lib/rancher/k3s/data/.lock"
Mar 15 14:57:53 dev-k3s-worker-3 k3s[1396]: time="2024-03-15T14:57:53Z" level=info msg="Preparing data dir /var/lib/rancher/k3s/data/*****"
Mar 15 14:57:54 dev-k3s-worker-3 k3s[1396]: time="2024-03-15T14:57:54Z" level=info msg="Starting k3s agent v1.27.11+k3s1 (06d6bc80)"
Mar 15 14:57:54 dev-k3s-worker-3 k3s[1396]: time="2024-03-15T14:57:54Z" level=info msg="Adding server to load balancer k3s-agent-load-balancer: [2a01:4f8:c012:3140::1]:6443"
Mar 15 14:57:54 dev-k3s-worker-3 k3s[1396]: time="2024-03-15T14:57:54Z" level=info msg="Running load balancer k3s-agent-load-balancer 127.0.0.1:6444 -> [[2a01:4f8:c012:3140::1]:6443] [default: [2a01:4f8:c012:3140::1]:6443]"
Mar 15 14:57:54 dev-k3s-worker-3 k3s[1396]: time="2024-03-15T14:57:54Z" level=warning msg="Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to enable Cluster CA validation."
Mar 15 14:57:55 dev-k3s-worker-3 k3s[1396]: time="2024-03-15T14:57:55Z" level=info msg="Using private registry config file at /etc/rancher/k3s/registries.yaml"
Mar 15 14:57:55 dev-k3s-worker-3 k3s[1396]: time="2024-03-15T14:57:55Z" level=info msg="Module overlay was already loaded"
Mar 15 14:57:55 dev-k3s-worker-3 k3s[1396]: time="2024-03-15T14:57:55Z" level=info msg="Module br_netfilter was already loaded"
Mar 15 14:57:55 dev-k3s-worker-3 k3s[1396]: time="2024-03-15T14:57:55Z" level=info msg="Set sysctl 'net/ipv4/conf/all/forwarding' to 1"
Mar 15 14:57:55 dev-k3s-worker-3 k3s[1396]: time="2024-03-15T14:57:55Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_max' to 131072"
Mar 15 14:57:55 dev-k3s-worker-3 k3s[1396]: time="2024-03-15T14:57:55Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400"
Mar 15 14:57:55 dev-k3s-worker-3 k3s[1396]: time="2024-03-15T14:57:55Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600"
Mar 15 14:57:55 dev-k3s-worker-3 k3s[1396]: time="2024-03-15T14:57:55Z" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
Mar 15 14:57:55 dev-k3s-worker-3 k3s[1396]: time="2024-03-15T14:57:55Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
Mar 15 14:57:56 dev-k3s-worker-3 k3s[1396]: time="2024-03-15T14:57:56Z" level=info msg="containerd is now running"
... bla bla bla
This successful addition clearly contrasts with the failure encountered with servers from other providers, highlighting a unique issue that seems to occur outside the Hetzner environment.
I expected the non-Hetzner servers to be added to the cluster as worker nodes without issues, similar to how Hetzner servers were added. I tried adding them using the standard command provided for Rancher cluster expansion, assuming network connectivity was not an issue based on my tests. Instead, I encountered a persistent "Waiting for agent to check in and apply initial plan" message and saw errors related to CA certificate acquisition in the logs.
