I have a k8s cluster configured with calico as networking system. I'm running slightly customized versions of helm elastic/elasticsearch and elastic/kibana with security enabled. For security, I use Let's Encrypt certificates. When starting kibana, the connection to the elasticsearch instance fails with this error
{"type":"log","@timestamp":"2021-06-01T13:09:55+00:00","tags":["debug","elasticsearch","query","data"],"pid":952,"message":"[ConnectionError]: unable to get issuer certificate"}
I can resolve this problem, by disabling certificate verification in kibana. Can anyone see, why its failing?
keystore creation
cat cert1.pem > store.pem
cat privkey1.pem >> store.pem
cat chain1.pem >> store.pem
cat fullchain1.pem >> store.pem
openssl pkcs12 -export -in store.pem -out keystore.pkcs12
values_elastic.yaml
replicas: 1
minimumMasterNodes: 1
esConfig:
elasticsearch.yml: |
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs-gen/keystore.pkcs12
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs-gen/keystore.pkcs12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs-gen/keystore.pkcs12
xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs-gen/keystore.pkcs12
xpack.security.enabled: true
extraEnvs:
- name: ELASTIC_PASSWORD
valueFrom:
secretKeyRef:
name: elastic-credentials
key: password
- name: ELASTIC_USERNAME
valueFrom:
secretKeyRef:
name: elastic-credentials
key: username
secretMounts:
- name: elastic-certificates
secretName: elastic-certificates
path: /usr/share/elasticsearch/config/certs-gen/
protocol: https
service:
labels: {}
labelsHeadless: {}
type: NodePort
nodePort: 30001
annotations: {}
httpPortName: http
transportPortName: transport
loadBalancerIP: ""
loadBalancerSourceRanges: []
externalTrafficPolicy: ""
clusterHealthCheckParams: "wait_for_status=yellow&timeout=1s"
values_kibana.yaml
elasticsearchHosts: "redacted its a TLD with appropriate port"
extraEnvs:
- name: "NODE_OPTIONS"
value: "--max-old-space-size=1800"
- name: 'ELASTICSEARCH_USERNAME'
valueFrom:
secretKeyRef:
name: elastic-credentials
key: username
- name: 'ELASTICSEARCH_PASSWORD'
valueFrom:
secretKeyRef:
name: elastic-credentials
key: password
- name: 'KIBANA_ENCRYPTION_KEY'
valueFrom:
secretKeyRef:
name: kibana
key: encryptionkey
- name: 'LOGGING_VERBOSE'
value: "true"
secretMounts:
- name: elastic-certificates
secretName: elastic-certificates
path: /usr/share/kibana/config/certs-gen/
kibanaConfig:
kibana.yml: |
server.ssl:
enabled: true
key: /usr/share/kibana/config/certs-gen/privkey1.pem
certificate: /usr/share/kibana/config/certs-gen/fullchain1.pem
elasticsearch.ssl:
certificateAuthorities: /usr/share/kibana/config/certs-gen/fullchain1.pem
verificationMode: certificate
xpack.reporting.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
xpack.security.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
xpack.encryptedSavedObjects.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
protocol: https
service:
type: NodePort
loadBalancerIP: ""
port: 5601
nodePort: 30002
labels: {}
annotations: {}
# cloud.google.com/load-balancer-type: "Internal"
# service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
# service.beta.kubernetes.io/azure-load-balancer-internal: "true"
# service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
# service.beta.kubernetes.io/cce-load-balancer-internal-vpc: "true"
loadBalancerSourceRanges: []
# 0.0.0.0/0
httpPortName: HTTP
kubectl get pv,pvc,nodes,pods,svc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/elk-data 30Gi RWO Retain Bound default/elasticsearch-master-elasticsearch-master-0 40m
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/elasticsearch-master-elasticsearch-master-0 Bound elk-data 30Gi RWO 32m
NAME STATUS ROLES AGE VERSION
node/kubeloadbalan Ready control-plane,master 28h v1.21.1
NAME READY STATUS RESTARTS AGE
pod/elasticsearch-master-0 1/1 Running 0 13m
pod/kibana-kibana-7fdbd7c66d-bg5xb 0/1 Running 0 7m1s
pod/nginx-deployment-868c6bb874-tsbg4 1/1 Running 0 40m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/elasticsearch-master NodePort 10.97.47.66 <none> 9200:30001/TCP,9300:32185/TCP 13m
service/elasticsearch-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 13m
service/kibana-kibana NodePort 10.101.78.90 <none> 5601:30002/TCP 7m1s
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 28h
service/nginx-service NodePort 10.100.231.27 <none> 80:30000/TCP 40m
From inside the container:
kubectl exec pod/kibana-kibana-7fdbd7c66d-bg5xb -it bash
curl -k -u redacted:redacted https://redacted:30001
{
"name" : "elasticsearch-master-0",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "iXEuqB7iQ9abptIZ_Gp1yg",
"version" : {
"number" : "7.13.0",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "5ca8591c6fcdb1260ce95b08a8e023559635c6f3",
"build_date" : "2021-05-19T22:22:26.081971330Z",
"build_snapshot" : false,
"lucene_version" : "8.8.2",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
curl -u redacted:redacted https://redacted:30001
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
That shows, that even curl cannot verify legitimacy from inside the container. That might indicate that the problem is also located on elasticsearch. The full Kibana log is a complete mess due to increased verbosity. I can post it on request.
I sank days into that. I'm burnt out and don't know where to continue.