0

I'm facing this kubelet error on my k8s cluster running CentOS 7. This error surfaced after I reboot the cluster nodes recently. I have not faced similar issue previously I reboot the machines.

I tried running swapoff -a to disable swap but it does not fix the problem.

Here's the systemctl status kubelet -l log:

[root@test-master ~]# systemctl status kubelet -l
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Mon 2022-05-30 13:59:51 +08; 822ms ago
     Docs: https://kubernetes.io/docs/
  Process: 9325 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
 Main PID: 9325 (code=exited, status=1/FAILURE)

May 30 13:59:51 test-master kubelet[9325]: Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_RC4_128_SHA. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
May 30 13:59:51 test-master kubelet[9325]: --tls-min-version string                                   Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13 (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
May 30 13:59:51 test-master kubelet[9325]: --tls-private-key-file string                              File containing x509 private key matching --tls-cert-file. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
May 30 13:59:51 test-master kubelet[9325]: --topology-manager-policy string                           Topology Manager policy to use. Possible values: 'none', 'best-effort', 'restricted', 'single-numa-node'. (default "none") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
May 30 13:59:51 test-master kubelet[9325]: --topology-manager-scope string                            Scope to which topology hints applied. Topology Manager collects hints from Hint Providers and applies them to defined scope to ensure the pod admission. Possible values: 'container', 'pod'. (default "container") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
May 30 13:59:51 test-master kubelet[9325]: -v, --v Level                                                  number for the log level verbosity
May 30 13:59:51 test-master kubelet[9325]: --version version[=true]                                   Print version information and quit
May 30 13:59:51 test-master kubelet[9325]: --vmodule pattern=N,...                                    comma-separated list of pattern=N settings for file-filtered logging (only works for text log format)
May 30 13:59:51 test-master kubelet[9325]: --volume-plugin-dir string                                 The full path of the directory in which to search for additional third party volume plugins (default "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
May 30 13:59:51 test-master kubelet[9325]: --volume-stats-agg-period duration                         Specifies interval for kubelet to calculate and cache the volume disk usage for all pods and volumes.  To disable volume calculations, set to a negative number. (default 1m0s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)

This is the content of /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf mentioned in previous log:

[root@test-master ~]# cat  /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

I've also tried adding Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --fail-swap-on=false" to the file mentioned here along with systemctl daemon-reload and systemctl restart kubelet but it doesn't help.

Unfortunately, I'm not exactly sure about the server's version but it should be the same as my client version [v1.23.3].

[root@test-master ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server 10.17.98.171:6443 was refused - did you specify the right host or port?

Is there anyway I can salvage my cluster without having to reset the entire cluster? I'm hoping I can access my deployments previously running in the cluster.


Update:

I tried to find the error messages in the log using journalctl -fu kubelet and this is the closest thing I can find.

May 31 08:58:12 test-master systemd[1]: kubelet.service holdoff time over, scheduling restart.
May 31 08:58:12 test-master systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
May 31 08:58:12 test-master systemd[1]: Started kubelet: The Kubernetes Node Agent.
May 31 08:58:12 test-master kubelet[5280]: Error: failed to parse kubelet flag: unknown flag: --network-plugin

Also, here's my kubelet's version Kubernetes v1.24.1.

Seems like it's related to this issue on GitHub, https://github.com/kubernetes/website/issues/33640.

4
  • kubernetes API is unreachable, according to your "kubectl version". Check your kubernetes API service (or pod). Then check "journalctl -fu kubelet", which would be more telling than the systemctl status kubelet you shared so far. There would be lots of logs, look for some "E" prefix (errors).
    – SYN
    May 30, 2022 at 18:44
  • 1
    @SYN Thanks for the suggestion, that helped me narrow down the issue.
    – Joe S
    May 31, 2022 at 1:17
  • @JoeS , have you found a solution?
    – svecax
    Jul 20, 2022 at 20:02
  • @svecax Yea, apparently the issue was that I was using Docker as my CRI which was deprecated when I was upgrading from 1.23 to 1.24. Solved everything after switching my CRI to containerd.
    – Joe S
    Jul 25, 2022 at 1:44

3 Answers 3

0

If you are using containerd as your cri, then Make sure that cri is not included in the disabled_plugins list within /etc/containerd/config.toml

More Info from kubernetes page

0

The issue was that I was using Docker as the container runtime prior to Kubernetes v1.24. Docker as container runtime has became deprecated starting from v1.24. Everything is working after I switched the container runtime to containerd.

Here's what I followed to fix my issue: https://kubernetes.io/docs/tasks/administer-cluster/migrating-from-dockershim/change-runtime-containerd/

0

Im getting similar issues here. The other issue with Port 6443 was refused.

I also tried running swapoff -a, stop and retstart kubelet. Checked Network, kubernetes config file, etc.

#systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: activating (auto-restart) (Result: exit-code) since Wed 2023-05-24 13:07:50 UTC; 7s ago
       Docs: https://kubernetes.io/docs/home/
    Process: 111682 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
   Main PID: 111682 (code=exited, status=1/FAILURE)
#kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.27.1
Kustomize Version: v5.0.1
The connection to the server k8s-master.dcs2.local:6443 was refused - did you specify the right host or port?
2
  • Is this supposed to be a solution? If yes, it's hard to see it, you should improve the answer. If you have a problem yourself you should use the Ask question button to ask a new question. May 24, 2023 at 13:37
  • If you have a new question, please ask it by clicking the Ask Question button. Include a link to this question if it helps provide context. - From Review
    – HBruijn
    May 26, 2023 at 13:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .