0

I have cluster with 1 control panel and 2 nodes.

Istio is installed as Service Mesh.

I do request management via istio ingress.

I want it to automatically scale by sharing metrics between Kubernetes HPA and istio prometheus, but I couldn't.

My pods on kube-system

kube-system pods

root@ubuntu-master:~# kubectl get pods -n kube-system
NAME                                    READY   STATUS    RESTARTS        AGE
coredns-78fcd69978-pk69f                1/1     Running   1 (2d11h ago)   2d12h
coredns-78fcd69978-t5dkx                1/1     Running   1 (2d11h ago)   2d12h
etcd-ubuntu-master                      1/1     Running   1 (2d11h ago)   48d
kube-apiserver-ubuntu-master            1/1     Running   2 (2d11h ago)   48d
kube-controller-manager-ubuntu-master   1/1     Running   3 (2d11h ago)   48d
kube-proxy-72q2r                        1/1     Running   0               2d10h
kube-proxy-8qgr9                        1/1     Running   1 (2d11h ago)   48d
kube-proxy-t4wgr                        1/1     Running   0               2d10h
kube-scheduler-ubuntu-master            1/1     Running   3 (2d11h ago)   48d
metrics-server-84b4bfc7fb-h8gd2         1/1     Running   0               18h

My pods on istio-system

istio-system

root@ubuntu-master:~# kubectl get pods -n istio-system
NAME                                    READY   STATUS    RESTARTS   AGE
grafana-6ccd56f4b6-bw5md                1/1     Running   0          2d12h
istio-ingressgateway-57c665985b-wj5gr   1/1     Running   0          2d12h
istiod-78cc776c5b-qkr6b                 1/1     Running   0          2d12h
jaeger-5d44bc5c5d-db2pj                 1/1     Running   0          2d12h
kiali-79b86ff5bc-mj8bn                  1/1     Running   0          2d12h
prometheus-64fd8ccd65-22znf             2/2     Running   0          2d12h
prometheus-adapter-6d9c6c8fdf-lxfbp     1/1     Running   0          17h

Prometheus UI result :

Prometheus UI result

Metrics server response;

Metrics server response

root@ubuntu-master:~# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"custom.metrics.k8s.io/v1beta1","resources":[{"name":"namespaces/network_transmit_packets_dropped","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/processes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/sockets","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/spec_memory_swap_limit_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"jobs.batch/kiali_single_validation_processing_duration_seconds_sum","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"jobs.batch/kiali_validation_processing_duration_seconds_count","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/kubelet_container_log_filesystem_used_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/cpu_system","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_user","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_reads_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"services/kiali_validation_processing_duration_seconds_sum","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},......

here in my HPA definition

istio kubernetes hpa

root@ubuntu-master:~# kubectl describe hpa html-pdf-v2-hpa -n html-pdf-v2
Name:                                                            html-pdf-v2-hpa
Namespace:                                                       html-pdf-v2
Labels:                                                          <none>
Annotations:                                                     metric-config.object.istio-requests-total.prometheus/query:
                                                                   sum(
                                                                     rate(
                                                                       istio_requests_total{
                                                                         destination_workload="html-pdf-deploy",
                                                                         destination_workload_namespace="html-pdf-v2"
                                                                       }[1m]
                                                                     )
                                                                   ) /
                                                                   count(
                                                                     count(
                                                                       container_memory_usage_bytes{
                                                                         namespace="html-pdf-v2",
                                                                         pod=~"html-pdf-deploy.*"
                                                                       }
                                                                     ) by (pod)
                                                                   )
CreationTimestamp:                                               Wed, 19 Jan 2022 18:11:14 +0300
Reference:                                                       Deployment/html-pdf-deploy
Metrics:                                                         ( current / target )
  "istio-requests-total" on Pod/html-pdf-deploy (target value):  <unknown> / 10
Min replicas:                                                    1
Max replicas:                                                    10
Deployment pods:                                                 5 current / 0 desired
Conditions:
  Type           Status  Reason                 Message
  ----           ------  ------                 -------
  AbleToScale    True    SucceededGetScale      the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetObjectMetric  the HPA was unable to compute the replica count: unable to get metric istio-requests-total: Pod on html-pdf-v2 html-pdf-deploy/unable to fetch metrics from custom metrics API: the server could not find the metric istio-requests-total for pods
Events:
  Type     Reason                 Age                   From                       Message
  ----     ------                 ----                  ----                       -------
  Warning  FailedGetObjectMetric  55s (x4255 over 17h)  horizontal-pod-autoscaler  unable to get metric istio-requests-total: Pod on html-pdf-v2 html-pdf-deploy/unable to fetch metrics from custom metrics API: the server could not find the metric istio-requests-total for pods

kubectl top pods result

root@ubuntu-master:~# kubectl top pods -n istio-system
NAME                                    CPU(cores)   MEMORY(bytes)
grafana-6ccd56f4b6-bw5md                2m           39Mi
istio-ingressgateway-57c665985b-wj5gr   10m          85Mi
istiod-78cc776c5b-qkr6b                 7m           61Mi
jaeger-5d44bc5c5d-db2pj                 3m           809Mi
kiali-79b86ff5bc-mj8bn                  4m           1125Mi
prometheus-64fd8ccd65-22znf             34m          744Mi
prometheus-adapter-6d9c6c8fdf-lxfbp     40m          76Mi

HPA Yaml.

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: html-pdf-v2-hpa
  namespace: html-pdf-v2
  annotations:
    metric-config.object.istio-requests-total.prometheus/query: |
      sum(
        rate(
          istio_requests_total{
            destination_workload="html-pdf-deploy",
            destination_workload_namespace="html-pdf-v2"
          }[1m]
        )
      ) /
      count(
        count(
          container_memory_usage_bytes{
            namespace="html-pdf-v2",
            pod=~"html-pdf-deploy.*"
          }
        ) by (pod)
      )
spec:
  maxReplicas: 10
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: html-pdf-deploy
  metrics:
    - type: Object
      object:
        metricName: istio-requests-total
        target:
          apiVersion: v1
          kind: Pod
          name: html-pdf-deploy
        targetValue: 10

I have concerns about where I went wrong or if I was walking on the right path.

First post I'm excited for answers. I hope I explained myself correctly.

Thanks

14
  • Could you describe your k8s monitoring configuration? How is your prometheus configured? Do you have a dedicated prometheus installation or the istio embeded one? Do you have the prometheus adapter and metrics server installed in your cluster?
    – c4f4t0r
    Jan 20 at 8:33
  • 1
    Don't post text information as image links.
    – AlexD
    Jan 20 at 8:33
  • @AlexD I can't add pictures because; "You need at least 10 reputation to post images."
    – Orgenus
    Jan 20 at 8:43
  • Add the text information as a text.
    – AlexD
    Jan 20 at 8:44
  • @c4f4t0r Installed with Prometheus istio's settings here. I installed the metric server with helm and set the tls here. I installed it with Prometheus adapter helm here.
    – Orgenus
    Jan 20 at 8:50

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.