0

I have a fresh kubernetes cluster (version 1.18) deployed using Rancher on a fresh Ubuntu Server 20.04. The problem I'm facing is that each https request made from any pod is failing with the following message:

curl -v https://pypi.org/simple/pip/
*   Trying 44.227.65.245...
* Connected to pypi.org (44.227.65.245) port 443 (#0)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* found 692 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* gnutls_handshake() failed: Handshake failed
* Closing connection 0
curl: (35) gnutls_handshake() failed: Handshake failed

I've tested it using

kubectl create deployment hello-node --image=k8s.gcr.io/echoserver:1.4
kubectl exec -it <pod_id> bash

I tried the same thing via ssh on the host system and everything was working properly. I also tried running standalone docker container (on the same machine) with the same image as pod (docker run -it --rm k8s.gcr.io/echoserver:1.4 bash) and there were no problems here as well. So I'm guessing that this is something related to my kubernetes setup.

It happend only with https requests - http requests are working properly.

I tried to remove the cluster and prepare it from scratch but this didn't help.

UPDATE:

I'm receiving two types of messages, depending on the server I'm trying to connect:

root@hello-node-7bf657c596-smr2h:/# openssl s_client -connect google.com:443 -debug
CONNECTED(00000003)
write to 0xda15b0 [0xda1630] (305 bytes => 305 (0x131))
0000 - 16 03 01 01 2c 01 00 01-28 03 03 4e c0 cf d9 62   ....,...(..N...b
0010 - 58 6e 88 7b 28 7f e8 c1-62 c6 8f 23 60 86 b0 53   Xn.{(...b..#`..S
0020 - df 43 cd a7 58 52 bc 59-6a ae bf 00 00 aa c0 30   .C..XR.Yj......0
0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a5 00 a3 00 a1   .,.(.$..........
0040 - 00 9f 00 6b 00 6a 00 69-00 68 00 39 00 38 00 37   ...k.j.i.h.9.8.7
0050 - 00 36 00 88 00 87 00 86-00 85 c0 32 c0 2e c0 2a   .6.........2...*
0060 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f   .&.......=.5.../
0070 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a4 00 a2 00 a0   .+.'.#..........
0080 - 00 9e 00 67 00 40 00 3f-00 3e 00 33 00 32 00 31   ...g.@.?.>.3.2.1
0090 - 00 30 00 9a 00 99 00 98-00 97 00 45 00 44 00 43   .0.........E.D.C
00a0 - 00 42 c0 31 c0 2d c0 29-c0 25 c0 0e c0 04 00 9c   .B.1.-.).%......
00b0 - 00 3c 00 2f 00 96 00 41-c0 11 c0 07 c0 0c c0 02   .<./...A........
00c0 - 00 05 00 04 c0 12 c0 08-00 16 00 13 00 10 00 0d   ................
00d0 - c0 0d c0 03 00 0a 00 ff-01 00 00 55 00 0b 00 04   ...........U....
00e0 - 03 00 01 02 00 0a 00 1c-00 1a 00 17 00 19 00 1c   ................
00f0 - 00 1b 00 18 00 1a 00 16-00 0e 00 0d 00 0b 00 0c   ................
0100 - 00 09 00 0a 00 23 00 00-00 0d 00 20 00 1e 06 01   .....#..... ....
0110 - 06 02 06 03 05 01 05 02-05 03 04 01 04 02 04 03   ................
0120 - 03 01 03 02 03 03 02 01-02 02 02 03 00 0f 00 01   ................
0130 - 01                                                .
read from 0xda15b0 [0xda6b90] (7 bytes => 7 (0x7))
0000 - 15 00 00 00 02 02 28                              ......(
140080050284184:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:794:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 305 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1600154660
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

and

root@hello-node-7bf657c596-smr2h:/# openssl s_client -connect pypi.org:443 -debug
CONNECTED(00000003)
write to 0x24645b0 [0x2464630] (305 bytes => 305 (0x131))
0000 - 16 03 01 01 2c 01 00 01-28 03 03 0c 2b bf dc 71   ....,...(...+..q
0010 - d2 41 56 f3 40 e4 c5 3f-62 88 cf 2a a7 76 e4 a4   .AV.@..?b..*.v..
0020 - 08 96 13 89 30 13 fa 75-c9 2d 32 00 00 aa c0 30   ....0..u.-2....0
0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a5 00 a3 00 a1   .,.(.$..........
0040 - 00 9f 00 6b 00 6a 00 69-00 68 00 39 00 38 00 37   ...k.j.i.h.9.8.7
0050 - 00 36 00 88 00 87 00 86-00 85 c0 32 c0 2e c0 2a   .6.........2...*
0060 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f   .&.......=.5.../
0070 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a4 00 a2 00 a0   .+.'.#..........
0080 - 00 9e 00 67 00 40 00 3f-00 3e 00 33 00 32 00 31   ...g.@.?.>.3.2.1
0090 - 00 30 00 9a 00 99 00 98-00 97 00 45 00 44 00 43   .0.........E.D.C
00a0 - 00 42 c0 31 c0 2d c0 29-c0 25 c0 0e c0 04 00 9c   .B.1.-.).%......
00b0 - 00 3c 00 2f 00 96 00 41-c0 11 c0 07 c0 0c c0 02   .<./...A........
00c0 - 00 05 00 04 c0 12 c0 08-00 16 00 13 00 10 00 0d   ................
00d0 - c0 0d c0 03 00 0a 00 ff-01 00 00 55 00 0b 00 04   ...........U....
00e0 - 03 00 01 02 00 0a 00 1c-00 1a 00 17 00 19 00 1c   ................
00f0 - 00 1b 00 18 00 1a 00 16-00 0e 00 0d 00 0b 00 0c   ................
0100 - 00 09 00 0a 00 23 00 00-00 0d 00 20 00 1e 06 01   .....#..... ....
0110 - 06 02 06 03 05 01 05 02-05 03 04 01 04 02 04 03   ................
0120 - 03 01 03 02 03 03 02 01-02 02 02 03 00 0f 00 01   ................
0130 - 01                                                .
read from 0x24645b0 [0x2469b90] (7 bytes => 7 (0x7))
0000 - 15 03 03 00 02 02 28                              ......(
140284912154264:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:769:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 305 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1600154752
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

I found this two messages appearing together described here: https://blog.techstacks.com/2010/03/3-common-causes-of-unknown-ssl-protocol-errors-with-curl.html

The Destination Site Does Not Like the Cipher You could be trying to connect to the site using an ssl cipher that the site is configured to reject. For example, anonymous ciphers are typically disabled on ssl-encrypted sites that are customer-facing. (Many of us set a blanket rejection policy on any SSL-encrypted web site—regardless of it's purpose.) The following command string "can" also result in the curl (35) error:

curl --ciphers ADH-RC4-MD5 https://some_web_site.some_domain.com/ Unfortunately, the type of error response you can get from curl depends largely upon the ssl server. On some sites, you'll receive the Unknown SSL Protocol error but on my techstacks-tools site, I get:

curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

Another thing I dicovered is that I prepare the same environment on the DigitalOcean droplet and ther were no problems there.

My bare metal environment is not available from the world, it stands inside internal network - maybe this is somehow related?

I also tried to setup k8s using microk8s but the problem still occurs so this is probably not related to rancher but only to my machine setup.

YAML of the pod:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/podIP: 10.42.0.8/32
    cni.projectcalico.org/podIPs: 10.42.0.8/32
  creationTimestamp: "2020-09-14T19:51:57Z"
  generateName: hello-node-7bf657c596-
  labels:
    app: hello-node
    pod-template-hash: 7bf657c596
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:generateName: {}
        f:labels:
          .: {}
          f:app: {}
          f:pod-template-hash: {}
        f:ownerReferences:
          .: {}
          k:{"uid":"dda770d0-3aa3-4fc9-97e1-c929fc3629e9"}:
            .: {}
            f:apiVersion: {}
            f:blockOwnerDeletion: {}
            f:controller: {}
            f:kind: {}
            f:name: {}
            f:uid: {}
      f:spec:
        f:containers:
          k:{"name":"echoserver"}:
            .: {}
            f:image: {}
            f:imagePullPolicy: {}
            f:name: {}
            f:resources: {}
            f:terminationMessagePath: {}
            f:terminationMessagePolicy: {}
        f:dnsPolicy: {}
        f:enableServiceLinks: {}
        f:restartPolicy: {}
        f:schedulerName: {}
        f:securityContext: {}
        f:terminationGracePeriodSeconds: {}
    manager: kube-controller-manager
    operation: Update
    time: "2020-09-14T19:51:57Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
          .: {}
          k:{"type":"PodScheduled"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
    manager: kube-scheduler
    operation: Update
    time: "2020-09-14T19:51:57Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:cni.projectcalico.org/podIP: {}
          f:cni.projectcalico.org/podIPs: {}
    manager: calico
    operation: Update
    time: "2020-09-14T19:52:01Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
          k:{"type":"ContainersReady"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
          k:{"type":"Initialized"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
          k:{"type":"Ready"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
        f:containerStatuses: {}
        f:hostIP: {}
        f:phase: {}
        f:podIP: {}
        f:podIPs:
          .: {}
          k:{"ip":"10.42.0.8"}:
            .: {}
            f:ip: {}
        f:startTime: {}
    manager: kubelet
    operation: Update
    time: "2020-09-14T19:52:01Z"
  name: hello-node-7bf657c596-smr2h
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: hello-node-7bf657c596
    uid: dda770d0-3aa3-4fc9-97e1-c929fc3629e9
  resourceVersion: "3468"
  selfLink: /api/v1/namespaces/default/pods/hello-node-7bf657c596-smr2h
  uid: cf37b215-8269-42ce-9e70-750f9f862cac
spec:
  containers:
  - image: k8s.gcr.io/echoserver:1.4
    imagePullPolicy: IfNotPresent
    name: echoserver
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-9jlms
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: gepard
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: default-token-9jlms
    secret:
      defaultMode: 420
      secretName: default-token-9jlms
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-09-14T19:52:00Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2020-09-14T19:52:01Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2020-09-14T19:52:01Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2020-09-14T19:52:00Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://2d2ddbf42fc4e6634a782c7036cf4a9a1d9f50a3d847bb5444932c701cd8186e
    image: k8s.gcr.io/echoserver:1.4
    imageID: docker-pullable://k8s.gcr.io/echoserver@sha256:5d99aa1120524c801bc8c1a7077e8f5ec122ba16b6dda1a5d3826057f67b9bcb
    lastState: {}
    name: echoserver
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2020-09-14T19:52:01Z"
  hostIP: 10.0.0.3
  phase: Running
  podIP: 10.42.0.8
  podIPs:
  - ip: 10.42.0.8
  qosClass: BestEffort
  startTime: "2020-09-14T19:52:00Z"

Firewall is disabled at the moment.

  • Any chance that you are using Proxy in your use case? – Wytrzymały Wiktor Sep 11 at 8:17
  • The only "proxy" is probably ingress but I'm not sure if it's participating in outbound traffic as well – John Sep 11 at 11:12
  • You will have to edit your question and include your Pod's YAML in order for anyone to help you, although I find "with the same image as pod and there were no problems" extraordinarily suspicious – mdaniel Sep 12 at 2:53
  • I've added commands used for creating deployment and creating a docker container – John Sep 13 at 7:47
  • @John This is not exactly what mdaniel has asked for. We need your Pod's YAML. Also, what is your firewall configuration (if present)? – Wytrzymały Wiktor Sep 15 at 11:10
1

The OP chose not to post an answer, but did say that they solved the problem:

The problem was that there was were some strange domains added in the search part of /etc/resolve.conf on the host. And when I was trying to make a https request I have been redirected by DNS to some other server responding with plain http.

Correcting the DNS settings solved the problem.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.