Skip to main content

Questions tagged [prometheus]

The Prometheus monitoring system.

Filter by
Sorted by
Tagged with
0 votes
0 answers
33 views

Prometheus node exporter as a side car container

We are trying to run node-exporter as a side-car container in kubernetes deployment. We have multiple filesystem devices, but in metrics we are getting only for single device filesystem metrics - name:...
karthik's user avatar
  • 101
0 votes
1 answer
78 views

Monitor HTTP request data for kubernetes microservice application

I have deployed a Java application in a Kubernetes cluster and am trying to figure out how to monitor it. So far I've managed to export logs, JVM metrics and traces through the Java open-telemetry ...
barmanthewise's user avatar
0 votes
0 answers
51 views

Node Exporter: couldn't get dbus connection

I use helm chart node_exporter. In this values.yaml https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus-node-exporter/values.yaml I only changed these lines hostRootFsMount:...
Maksim's user avatar
  • 113
0 votes
0 answers
59 views

How to monitor multiple kubernetes clusters using grafana and prometheus from a separate cluster?

I have three kubernetes clusters. (monitoring, one, two). monitoring cluster is dedicated for monitoring tools. So on monitoring cluster, I will install prometheus and grafana. On other cluster should ...
Mysterio's user avatar
0 votes
0 answers
51 views

Kubelet/Cadvisor on GKE not exporting container_fs_* metrics for attached volumes

In our GKE 1.27.12 cluster, we run a couple of stateful workloads using GCP Volumes, e.g. using this storage class: apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: pd-ssd ...
antaxify's user avatar
  • 123
0 votes
0 answers
33 views

Error configuring node-exporter DaemonSet scraping for Prometheus on kubernetes - service account token missing

I have set up a node-exporter DaemonSet on kubernetes as well as a service that points to these node-exporter pods IPs (I followed this tutorial). When I run kubectl get endpoints -n monitoring, I ...
Paula Gouveia's user avatar
0 votes
1 answer
58 views

SOLVED - Error configuring node-exporter DaemonSet scraping for Prometheus on kubernetes

I am posting the following question already solved, because I've mistakenly posted in on StackOverflow and therefore wanted to share it here so it can be properly found by the community and hopefully ...
Paula Gouveia's user avatar
0 votes
1 answer
57 views

When to use Thanos/Cortex over Vanilla Prometheus?

Generally when I look around or try to understand the HA setup for Prometheus the most common search results are Cortex & Thanos. I've also seen M3 and Victoriametrics on the list. What I ...
Vipin Menon's user avatar
0 votes
0 answers
30 views

ServiceMonitor Relabeling EC2 Instance Tags Not Working in Kube-Prometheus-Stack

I'm using the kube-prometheus-stack Helm chart to deploy Prometheus and configure ServiceMonitors for monitoring EC2 instances with the Node Exporter. I have configured the ServiceMonitor to relabel ...
TomerA's user avatar
  • 1
0 votes
0 answers
26 views

Setting up Prometheus on Azure Kubernetes Cluster

I am setting up Prometheus on a production aks cluster. The app deployment on this cluster is exposed using ngnix ingress behind a load balancer and a firewall device. How do I access Prometheus ...
sakshi's user avatar
  • 1
0 votes
0 answers
22 views

“Annotations.runbook” doesn't generate a link in a Slack message

In my k8s cluster, deploy kube-prometheus-stack, it is configured to send alert manager notifications to Slack. For some unknown reason, the Runbook icon is inactive and nothing happens when you click ...
User.354125's user avatar
0 votes
1 answer
116 views

Prometheus import old data using promtool error

Follow up with similar issue faced by other: promtool tsdb create-blocks-from openmetrics doesn't show any data in Prometheus I am facing a similar issue. I have a scrape API endpoint which ...
Nazar Muhammad's user avatar
0 votes
0 answers
156 views

Prometheus default storage retention time

I started Prometheus (v.2.51.2) as a container like appended and expected the flag "storage.tsdb.retention.time to be set to "15d" as default, but it's default is "0s", whicht ...
dan_0's user avatar
  • 131
0 votes
1 answer
190 views

How do we configure prometheus server to scrape metrics from a pod with Istio sidecar proxy?

A service pod is running with Istio sidecar container and is MTLS enabled. How do we define a service monitor to scrape metrics from this service ? Do we need to update the Prometheus server for the ...
Nipun Talukdar's user avatar
0 votes
0 answers
45 views

kubelet_volume_* metrics not showing all PVCs

I use Prometheus to monitor my Kubernetes cluster. I have 42 PVCs Bound to 42 PVs. For some reason, kubelet_volume_* return information about only 29 of them. I looked everywhere and I could not find ...
Mohamed Aziz Tousli's user avatar
0 votes
0 answers
65 views

Trouble Configuring Zabbix Exporter with Prometheus for Grafana Dashboards

I'm currently attempting to set up Zabbix Exporter to fetch Zabbix metrics into Prometheus for visualization in Grafana dashboards. My goal is to utilize Grafana's features for creating insightful ...
Xoryy's user avatar
  • 1
0 votes
0 answers
22 views

Free/total and when free= 10% of total color turns red

I am using Grafana and I am trying to display the values of "Free space/Total size" as actual values rather than calculations. I would like to create a value mapping so that when the free ...
Xoryy's user avatar
  • 1
1 vote
0 answers
50 views

Prometheus CPU consumption after remote_write is enabled

I have problems figuring out why my Prometheus instance starts to chew a lot of CPU after I enable remote_write feature. I have deployed a prometheus and grafana from chart kube-prometheus-stack, ...
Antonio Soldo's user avatar
1 vote
1 answer
87 views

Grafana Agent in Docker runs perfectly for over a week and then suddenly stops working. When we check, the Data folder is gone

We have deployed the Grafana Agent using the official Docker image to send metrics of our server and containers to Grafana Cloud: docker run \ -v /etc/grafana-wal:/etc/agent/data \ -v /etc/grafana-...
Alvaro Bataller's user avatar
2 votes
2 answers
918 views

Prometheus/grafana : extract hostname from "node_uname_info" metric and create CPU, Memory usage grafana panels

I have populated prometheus.yml -> scrape_configs -> static_configs -> targets section using multiple IP addresses. There is no DNS functionality on prometheus server nor populating hostnames ...
Vijay Gharge's user avatar
0 votes
0 answers
95 views

Kubernetes monitoring with Prometheus

I need a little bit of help here. I have a Kubernetes cluster up and running and I have a dedicated machine for monitoring with Prometheus running on it. I already have node exporters running and ...
skylar's user avatar
  • 1
0 votes
1 answer
442 views

How to monitor multiple Kubernetes clusters using single Grafana?

I would like to use a single Grafana instance to monitor multiple Kubernetes cluster (pods resources consumption, rabbitmq queues info) provided by Prometheus. I have two Kubernetes cluster - one used ...
FN_'s user avatar
  • 273
1 vote
1 answer
615 views

How to use Grafana Alertmanager for Prometheus rules?

The company decided to centralize all alerting on Grafana (Unified Alerting). Alerts are created through the Grafana UI and the internal Grafana Alertmanager is used for routing alerts to the contact ...
user avatar
0 votes
0 answers
118 views

Monitor RabbitMQ with Prometheus outside kubernetes cluster

I'm trying to monitor a RabbitMQ set up in a kubernetes cluster by the bitnami helm chart with 3 pods from an Prometheus outside the cluster and I don't understand how that should be possible. On the ...
BMaehr's user avatar
  • 1
1 vote
1 answer
413 views

Prometheus Blackbox http_2xx returning 403 since webserver migrated to StackCP

I had working Prometheus Blackbox Exporter http_2xx checks monitoring various web servers. Then the web hosting provider migrated from cPanel to Stack CP. Since then all the http_2xx just return 403 (...
codlord's user avatar
  • 145
3 votes
1 answer
584 views

How to share docker metrics with Prometheus using Ubuntu server Snap Docker Service

I am using Ubuntu server with a Docker Service installed from the Server Setup (snap) and Prometheus running in a docker container. Server: Distributor ID: Ubuntu Description: Ubuntu 22.04.3 LTS ...
Hagbart Celine's user avatar
0 votes
1 answer
3k views

Kubernetes - Find per core statistics for the pod

I would want to find the per core usage statistics for my Kubernetes pod. In my Linux host/in the Kubernetes node, I use mpstat to find the statistics like below. In my case, I assign 2.5 CPUs to the ...
Vishal Raghavan's user avatar
0 votes
0 answers
193 views

Prometheus can't scrape custom metrics

I'm trying to set up Prometheus to scrape some custom metrics exposed by a java service with the java lib Prometheus Micrometer. I have no error whatsoever in prometheus log except for proto: bad ...
Ror's user avatar
  • 343
0 votes
1 answer
395 views

Use let's encrypt certificates with the prometheus node exporter

I'm trying to configure a prometheus node exporter with HTTPS using the website certificates generated by let's encrypt. There is certificates with permissions default let's encrypt permission But I ...
Zack's user avatar
  • 103
0 votes
0 answers
214 views

How to track PersistentVolume usage with prometheus for non gp2 storage class?

I am running a kubenetes cluster in an ubuntu node. I have created a persistent volume (storage class: openebs-hostpath) and corresponding claim. I want to track how much of the claim is being used by ...
sachinks's user avatar
1 vote
1 answer
300 views

Sending Prometheus data to local and remote storage

Having a PI running to monitor internet connectivity locally is good, but it also comes with the drawback of having limited storage, especially when other stuff is running on it, too. I am looking for ...
func0der's user avatar
  • 164
0 votes
0 answers
812 views

SNMP scraping returned HTTP status 500 Internal Server Error on Prometheus

I installed Prometheus and SNMP exporter on Ubuntu Server, then configuring for scraping with /etc/prometheus/prometheus.yml - job_name: 'Mikrotik' static_configs: - targets: - 10.10.106.210 ...
Adelaide Underwood's user avatar
0 votes
2 answers
208 views

How do I get the boot time for a node given the instance information from dockerswarm_sd_config

I am trying to get the up time from node exporter but I want to have it driven by the nodes provided by dockerswarm_sd - job_name: dockernodes scrape_interval: 5s dockerswarm_sd_configs: - host: ...
Archimedes Trajano's user avatar
0 votes
0 answers
34 views

Can Prometheus scale horizontally with a single data source?

I'm trying to run Prometheus on an unrealistic system (a Raspberry Pi) and I find that it keeps on losing connectivity which I presume is load related. I was wondering if it can be scaled ...
Archimedes Trajano's user avatar
0 votes
1 answer
170 views

Puppet syntax issue when configuring a Prometheus scrape on an SNMP exporter

Some context I have an SNMP exporter running as a service on a server which allows me to access the data of some PDU using the following URL: http://my.host.name:9116/snmp?target=10.0.1.200&auth=...
Patrick's user avatar
  • 105
0 votes
0 answers
340 views

Auto-Instrumentation of application using OpenTelemetry

I have an AKS cluster where I am running a test Python-Django based web application. I also have Grafana and Prometheus configured. I need to use OpenTelemetry to get the metrics data from the test ...
arjunbnair's user avatar
0 votes
0 answers
253 views

OpenTelemetry Collector Data not being fetched by Prometheus in Grafana

I have a requirement where I have some container workload in Azure AKS cluster and I need to use OpenTelemetry to gather data like metrics, logs and traces. I also have Grafana as the visualisation ...
arjunbnair's user avatar
0 votes
0 answers
183 views

OpenTelemetry K8s Operator Collector - Exporter Configuration for Prometheus

I have some container workloads in Azure AKS cluster. I need to use OpenTelemetry to get the metrics, logs and trace data from the container workload and get it collected by the OTEL collector. I have ...
arjunbnair's user avatar
0 votes
2 answers
1k views

How can i monitor Openstack with prometheus and grafana

I installed openstack with devstack on Ubuntu 20 and installed prometheus and grafana with node exporter and it showed on grafana no issues, but how to monitor openstack components Nova.... There is a ...
Hamzad's user avatar
  • 3
0 votes
0 answers
181 views

nginx - can't reach status endpoint on localhost

I try to setup the nginx exporter for prometheus but have difficulties with the proper configuration of my nginx. I have a default server block that redirects all requests from port 80 to 443. So all ...
andymel's user avatar
  • 211
0 votes
0 answers
185 views

I'm trying to expose a folder size in Node exporter but messes up. any suggestions?

This wasn't the case for a few months but suddenly my Node exporter in the server service messed up. Oct 02 10:13:01 nodefs01 node_exporter[117252]: time="2023-10-02T10:13:01Z" level=error ...
user3324696's user avatar
0 votes
0 answers
90 views

Write_Prometheus plugin with basic auth

I am testing the below Write_Prometheus plugin to send the cassandra metrics to prometheus server which is external machine but i see that all metrics are sending without any basic auth happening. so ...
rohit singhal's user avatar
0 votes
0 answers
139 views

Spot instance sometime slow down and loss connection

I have a system deployed in AWS EKS, sometimes spot instances metrics is down, and API call to these nodes are very slow. Here is my system: 1 EKS cluster 1 on-demand node group 1 Karpenter v0.29.2 ...
Tristan's user avatar
  • 21
0 votes
0 answers
352 views

Why Kube state metrics only shows metrics related to the namespace where it is running?

I have AWS EKS cluster with kube-state-metrics installed in a namespace called "monitoring". This installation is using service monitor and other components (see yaml files below). In this ...
Thiago Scodeler's user avatar
1 vote
0 answers
651 views

Prometheus does not start with WAL segment loaded

There is a Prometheus installed in Kubernetes Azure AKS with default storage StandardSSD. Prometheus version v2.47.0 The volume storage size is 1 Gb. Data retention is configured for 1 hour. Maximum ...
Michael Chudinov's user avatar
0 votes
0 answers
156 views

Correct way to setup a multinode LGTM stack

I have a 4 nodes clusters: MonitoringCenter, hosting a Grafana, connected to all Prometheus and Loki instances + local AlertManager a Prometheus scraping local NodeExporter/AlertManager/Loki/...
GlinesMome's user avatar
0 votes
0 answers
201 views

How to add configuration for fluent-plugin-prometheus in Fluentd deployed via Fleet in Rancher?

I'm using Rancher to manage my Kubernetes cluster and have added a logging system (cattle-logging-system) via Fleet. I now need to add monitoring for Fluentd using the fluent-plugin-prometheus. Here's ...
Maksim Karibov's user avatar
0 votes
0 answers
310 views

Thanos Receiver not deleting old data in Persistent Volume (PV) after retention is exceeded

I have set up Kube Prometheus Stack with Thanos on my Kubernetes cluster, and I'm using the Thanos Receiver instead of the sidecar approach. I have also configured the Thanos Compactor and Minio for ...
dasunNimantha's user avatar
0 votes
0 answers
223 views

Spike in Cadvisor container_network_receive_bytes_total Metric in a Kubernetes Cluster

Summary: I'm using Cadvisor with Prometheus in multiple Kubernetes (k8s) clusters to monitor network traffic usage. I utilize the container_network_receive_bytes_total metric in a query to calculate ...
Hesam Norin's user avatar
0 votes
0 answers
232 views

Prometheus Server Pod Suddenly Crashed (unexpected fault address 0x7f911b1795d4)

With traffic Prometheus server pod getting restarting with below error stack. This is happend when the live traffic of the system. But could not be able to reproduced with the load testing. Grafana ...
Sidath Weerasinghe's user avatar