2

We have tried to set up metrics server in our kubernetes cluster, and it keeps failing.

I am a bit unsure where I went wrong. The cluster has been set up and upgraded using kubeadm on existing hardware. I see that during a lot of operations kubernetes tries and fails to communicate with metrics-server.

Anyone has experienced this and/or can help me finding the cause of this?

Here some output from the metric-server logs:

I0201 09:20:32.016226       1 manager.go:150] ScrapeMetrics: time: 216.595261ms, nodes: 5, pods: 49
I0201 09:20:32.016257       1 manager.go:115] ...Storing metrics...
I0201 09:20:32.016319       1 manager.go:126] ...Cycle complete
E0201 09:20:32.596639       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:32.596839       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (615.212µs) 401 [[kubectl/v1.13.2 (linux/amd64) kubernetes/cff46ab] 10.46.0.0:44210]
E0201 09:20:32.636449       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:32.636590       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (460.541µs) 401 [[kubectl/v1.13.2 (linux/amd64) kubernetes/cff46ab] 10.46.0.0:44210]
I0201 09:20:37.552609       1 request.go:897] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0201 09:20:37.552813       1 round_trippers.go:386] curl -k -v -XPOST  -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: metrics-server/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJtZXRyaWNzLXNlcnZlci10b2tlbi10bTRmdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJtZXRyaWNzLXNlcnZlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImMyN2VjNDAxLTI1NjMtMTFlOS04Y2M2LTFjYzFkZWVhNzdjOCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTptZXRyaWNzLXNlcnZlciJ9.YF4IaGtM_IlRZ8Xzik3AEDnv6-Q4YQBamBjna_gLydhVehH4gmq_Y4y0Nrcqt4Ana9HwNcLx0jGV4GU-njUfzrb0uS9eKl2Eeh6bLTkwafKAv7cF8SwP0rBLuhIl6FDgwBU4d95MQAqOxvMdnlSquJmYOiuIT25OxD_wPJ2PYjdXbuxxSChvrLrtGwa5URbzNvN9deMWSugbz2B1knCu8YAlKPx31bUEa27YFCZIrtydRjY2E1Qzl8hkJiEuom8v_sRLTvnJyYcOU6ARWqwJT570JeubMO5_GcvnpVpmBmh8QFr8_BLTJJfiEleFNs9YmBgWIr3xDwjEBDmn5ndjrQ" 'https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews'
I0201 09:20:37.572204       1 round_trippers.go:405] POST https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 19 milliseconds
I0201 09:20:37.572235       1 round_trippers.go:411] Response Headers:
I0201 09:20:37.572245       1 round_trippers.go:414]     Content-Type: application/json
I0201 09:20:37.572254       1 round_trippers.go:414]     Content-Length: 260
I0201 09:20:37.572262       1 round_trippers.go:414]     Date: Fri, 01 Feb 2019 09:20:37 GMT
I0201 09:20:37.572323       1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0201 09:20:37.572465       1 authorization.go:73] Forbidden: "/", Reason: ""
I0201 09:20:37.572580       1 wrap.go:42] GET /: (20.227877ms) 403 [[Go-http-client/2.0] 10.46.0.0:44198]
I0201 09:20:39.404760       1 authorization.go:73] Forbidden: "/", Reason: ""
I0201 09:20:39.404908       1 wrap.go:42] GET /: (321.809µs) 403 [[Go-http-client/2.0] 10.46.0.0:44198]
I0201 09:20:39.451089       1 authorization.go:73] Forbidden: "/", Reason: ""
I0201 09:20:39.451212       1 wrap.go:42] GET /: (283.995µs) 403 [[Go-http-client/2.0] 10.46.0.0:44198]
E0201 09:20:40.708131       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:40.708327       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (544.441µs) 401 [[kube-controller-manager/v1.13.2 (linux/amd64) kubernetes/cff46ab/controller-discovery] 10.46.0.0:44210]
E0201 09:20:40.955975       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:40.956151       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (574.914µs) 401 [[kube-controller-manager/v1.13.2 (linux/amd64) kubernetes/cff46ab/system:serviceaccount:kube-system:generic-garbage-collector] 10.46.0.0:44210]
E0201 09:20:41.785405       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:41.785570       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (579.992µs) 401 [[kube-controller-manager/v1.13.2 (linux/amd64) kubernetes/cff46ab/system:serviceaccount:kube-system:generic-garbage-collector] 10.46.0.0:44210]
E0201 09:20:42.065074       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:42.065248       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (566.86µs) 401 [[kubectl/v1.13.2 (linux/amd64) kubernetes/cff46ab] 10.46.0.0:44210]
E0201 09:20:42.305102       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:42.305272       1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (552.597µs) 401 [[kubectl/v1.13.2 (linux/amd64) kubernetes/cff46ab] 10.46.0.0:44210]

And this from kube-apiserver logs:

I0201 09:22:14.652152       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0201 09:22:19.688846       1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0201 09:22:49.751772       1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0201 09:23:19.816917       1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0201 09:23:49.896396       1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
I0201 09:24:14.314774       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0201 09:24:14.317317       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 401, Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
, Header: map[Content-Length:[129] Date:[Fri, 01 Feb 2019 09:24:14 GMT] Content-Type:[application/json]]
I0201 09:24:14.317368       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0201 09:24:19.960927       1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0201 09:24:50.037553       1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
I0201 09:25:14.317811       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0201 09:25:14.320556       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 401, Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
, Header: map[Content-Length:[129] Date:[Fri, 01 Feb 2019 09:25:14 GMT] Content-Type:[application/json]]
I0201 09:25:14.320623       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0201 09:25:20.110375       1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0201 09:25:50.172368       1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
2

Following the https://github.com/kubernetes-incubator/metrics-server/issues/67, https://github.com/kubernetes-incubator/metrics-server/issues/146 and https://github.com/kubernetes-incubator/metrics-server/issues/131 you may want try use next solution:

For future readers scratching their heads: on a Kubernetes 1.13 cluster deployed with kubeadm, metrics server started working once I updated the deployment spec with the following:

 command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP 

(After that, give it a few minutes before kubectl top actually has enough data to show anything, though.)

Or at least try to modify metrics-server Deployment to

    command:
    - /metrics-server
    - --kubelet-insecure-tls
|improve this answer|||||
2

This issue has been reported on https://github.com/kubernetes/kubernetes/issues/69277 and further discussed on https://github.com/kubernetes/kubernetes/issues/61879. As stated, Multi-master setup the ca crt/key files should be generated outside and set in the /etc/kubernetes/pki/* folder so that Kubeadm can issue the server cert and client certs using the ca files. I hope this helps.

|improve this answer|||||
  • This sounds a lot like it is the problem I have, but how do I fix it? – Flyhard Feb 1 '19 at 14:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.