1

Background

Created a fresh Kubernetes cluster using kubeadm init --config /home/kube/kubeadmn-config.yaml --upload-certs and then joining the 2nd control plane node by running the below.

kubeadm join VIP:6443 --token <token> \
    --discovery-token-ca-cert-hash sha256:<hash> \
    --control-plane --certificate-key <key> \
    --v=5

Question

Is etcdctl commands supposed to come back with a return value? Either using the command directly or using the docker exec method shown below. I have these packages installed kubeadm, kubectl, kubelet, and docker.

Kubectl version: 1.20.1 OS: Ubuntu 18.04

Commands from the first node

Command

etcdctl cluster-health

Response

cluster may be unhealthy: failed to list members
Error:  client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:4001: connect: connection refused
; error #1: EOF

error #0: dial tcp 127.0.0.1:4001: connect: connection refused
error #1: EOF

Command

docker container ls | grep k8s_POD_etcd

Response

k8s_POD_etcd-<nodename>_kube-system_<docker container id>

Command

docker exec -it k8s_POD_etcd-<nodename>_kube-system_<docker container id> etcdctl --endpoints=https://<node ip>:2379 --key=/etc/kubernetes/pki/etcd/peer.key --cert=/etc/kubernetes/pki/etcd/peer.crt --cacert=/etc/kubernetes/pki/etcd/ca.crt member list

Response

OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "exec: \"etcdctl\": executable file not found in $PATH": unknown

EDIT

Upgraded to v3.2 etcdctl API

Command

etcdctl endpoint status

Response

Failed to get the status of endpoint 127.0.0.1:2379 (context deadline exceeded)
4
  • Hi PieDev, welcome to S.F. That 4001 port is the legacy one, used by etcd2 which is almost certainly not supported by k8s; I would guess it's either an ancient binary or is missing ETCDCTL_API=3 and the associated --endpoints (ETCDCTL_ENDPOINTS) values to point it to the modern :2379 port. I would further guess the etcd certs are volume mounted from /etc/kubernetes/pki/etcd on the host, and thus you don't need etcdctl to exist inside the docker image, you can use the system version – mdaniel Jan 24 at 3:59
  • @PieDev Any progress? – Wytrzymały Wiktor Jan 27 at 10:10
  • @mdaniel See the edit I made. I do see .crt and .key files in ls -l /etc/kubernetes/pki/etcd/. – PieDev Feb 1 at 3:20
  • @WytrzymałyWiktor See the edit I made. – PieDev Feb 1 at 3:20
0

The context deadline exceeded is an unclear error returned by grpc client when it can't establish the connection. If you want to see the exact error message you should set ETCDCTL_API=2 (more details on that can be found here).

The cert/key pairs in /etc/kubernetes/pki/etcd/ should look something like this:

# ls -l /etc/kubernetes/pki/etcd/
total 32
-rw-r--r--    1 root     root          1017 Nov 12 15:32 ca.crt
-rw-------    1 root     root          1679 Nov 12 15:32 ca.key
-rw-r--r--    1 root     root          1094 Nov 12 15:32 healthcheck-client.crt
-rw-------    1 root     root          1675 Nov 12 15:32 healthcheck-client.key
-rw-r--r--    1 root     root          1180 Nov 12 15:32 peer.crt
-rw-------    1 root     root          1675 Nov 12 15:32 peer.key
-rw-r--r--    1 root     root          1180 Nov 12 15:32 server.crt
-rw-------    1 root     root          1679 Nov 12 15:32 server.key

# etcdctl --version
etcdctl version: 3.3.1
API version: 2

# ETCDCTL_API=3 etcdctl snapshot save snapshot.db \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/server.crt \
  --key /etc/kubernetes/pki/etcd/server.key
Snapshot saved at snapshot.db

# ETCDCTL_API=3 etcdctl --write-out=table snapshot status snapshot.db
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| b9d500f7 |    72966 |       1194 |     4.9 MB |

Make sure that you apply the right cert/key pair. Also, this guide can help you out.

Note that etcd takes several certificate related configuration options, either through command-line flags or environment variables. The basic setup for it can be found here.

2
  • @PieDev Does this answer your question? – Wytrzymały Wiktor Feb 5 at 9:19
  • @PieDev If the above does not solve your issue than please edit your question and provide the output from kubectl get pods -n kube-system and kubectl -n kube-system describe pod etcd to verify if etcd is running in you k8s cluster. Also, what was your kubeadm init config in terms of the etcd? – Wytrzymały Wiktor Feb 9 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.