2

I have a very strange issue on google cloud platform with one of my K8S clusters.

Two days ago, everything worked like a charm. But yesterday every helm commands used to deploy on this cluster started to fail with:

Error from server: error dialing backend: No SSH tunnels currently open. Were the targets able to accept an ssh-key for user "gke-5fe81XXXXXXX"? I checked every single leads from the https://cloud.google.com/kubernetes-engine/docs/troubleshooting The kubectl logs, attach, exec, and port-forward commands hang page.

  • Cluster do have nodes (15 node on 3 pools)
  • Every pods is running and alive in the cluster and I tried to connect on everyone of them: same issue for every pod on every node
  • My network firewall have both default-allow-ssh and the gke-review-5fe81XXX-ssh that allow ssh from the master ip to all nodes
  • My project metadatas sshKeys contains the master ssh key and is not 32kb long
  • on my nodes, the .ssh/authorized_keys file contains the public key from master
  • I can ssh to nodes from my computer after adding my key to project metadatas, and between nodes

What is really annoying is that it happened last week for maybe few hours but then the issue vanished magically.

Thank you.

1

If you run kubectl get nodes -o wide do you see an ExternalIP for the nodes or <none>? If you see <none>, try restarting the nodes and see if that fixes your issue.

  • Hi, thanks for your answer, every node has an external ip and I can ssh on it. The only issue is between the master and nodes. My only way to fix this issue has been to recreate a cluster from scratch. – Matthieu P. Dec 3 '18 at 8:11
  • Are you checking for external IP via the GCP APIs? Or are you seeing it when using kubectl? – Robert Bailey Dec 3 '18 at 15:52
  • I was getting IPs by kubectl get nodes -o wide but they are the same in the gcloud board. – Matthieu P. Dec 4 '18 at 18:46
  • What version of GKE you are running? Did you recently upgraded your cluster? Just wondering whether the cluster version upgrade causing the issue due to possible flaws at particular cluster. – Rahi Dec 6 '18 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.