Questions tagged [prometheus]

The Prometheus monitoring system.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

PromQL range vector, but only if there's data from the start of the range? (for a new metric)

Prometheus can return all values for a time (e.g. api_counter[1h] for all values in the last hour). When the metric is new, there will not be 1 hour of data points. Is it possible to write a promql ...
user avatar
0 votes
2 answers
37 views

RHEL + can we improve disks performance by tuning kernel parameters?

we have Hadoop cluster and we are collection metrics collection data in order to investigate slowness behavior on spark applications after long investigation on our Hadoop cluster we noticed from ...
user avatar
0 votes
0 answers
11 views

Resource utilisation split in virtualisation platforms

I'm having a Hyper-V hypervisor and am running a CentOS 7 VM on top of it. The VM houses certain microbenchmarks (sysbench, stream, iozone and netperf) and I'm trying to study the effects (utilisation)...
user avatar
0 votes
0 answers
4 views

Does Prometheus efficiently compress series of identical samples?

Does Prometheus efficiently store and de-duplicate long repeats of identical sample values for a metric? I'm working with a system where some of the inputs change at a low frequency - say every 30 ...
user avatar
  • 10.4k
0 votes
1 answer
55 views

prometheus field remote_write not found in type config.plain

I need help using prometheus remote_write and remote_read to/from influxdb. I got error below: # promtool check config /etc/prometheus/prometheus.yml Checking /etc/prometheus/prometheus.yml FAILED: ...
user avatar
  • 1
1 vote
0 answers
155 views

Hostname instead of IPs for panel legends in Grafana with Prometheus metrics

I want to monitor different servers with Prometheus fed by exporters like node-exporter or postgres-exporter. The metrices should be presented in Grafana as dashboards and alerts. The exporters will ...
user avatar
  • 111
0 votes
0 answers
88 views

How to get Kafka consumer metrics with JMX exporter for Prometheus?

Kafka version: 3.1 I didn't find a way to get Kafka consumer metrics as described in Kafka documentation from my Kafka servers. Even if I leave the "rules" section empty in the config of the ...
user avatar
0 votes
1 answer
58 views

Disabling email for Prometheus Alertmanager based on severity

We are monitoring a HPC computing cluster using a combination of Prometheus, Alertmanager and Grafana. On our machines, things like SWAP memory filling up to essentially the limit happen frequently, ...
user avatar
  • 101
0 votes
1 answer
246 views

Alertmanager telegram config chat_id and cannot unmarshal errror

I am trying to configure alertmanager to send alerts to my telegram group. Following the configuration I have: global: resolve_timeout: 5m route: group_by: - job group_interval: 5m ...
user avatar
  • 11
0 votes
0 answers
27 views

prometheus alert for Kubernetes misfiring

On a K8s cluster I have Prometheus Operator and AlertManager running. I have this alert to catch incidents when a critical pod is down: - alert: KubernetesContainerMission-gslNotRunning expr: ...
user avatar
  • 1
0 votes
0 answers
17 views

Change retention time on prometheus

So I am running Prometheus on an Ubuntu server and started running it a while ago by just running sudo ./prometheus However, I wanted to append --storage.tsdb.retention.time=1y to the end of this ...
user avatar
0 votes
0 answers
28 views

Secure kubernetes monitoring in Prometheus using service discovery

I'm currently using Prometheus to scrape metrics from a /metrics endpoint on my pods. In order to identify the pods I'm using kubernetes service discovery which discovers the pods and targets them ...
user avatar
0 votes
1 answer
111 views

webhook MS Teams integration with Prometheus - request failed

I'm struggling with Microsoft Teams/Prometheus integration on K8s cluster. I used helm to start all components. I have correctly working Prometheus and Alertmanager. It seems that all works fine. ...
user avatar
0 votes
1 answer
83 views

Nginx reverse proxy when port is already in use?

i am trying to close port with basic authentication (for pushgateway of prometheus), so not a big specialist in nginx, so could someone please give me and advice where i am is wrong? I have 9091 port, ...
user avatar
0 votes
1 answer
321 views

Debug Prometheus Blackbox Exporter http_2xx probs

We are using the Prometheus Blackbox Exporter (blackbox_exporter, version 0.19.0) to check HTTP endpoints. Now we need to implement a check with the fail_if_body_matches_regexp. The check should ...
user avatar
  • 121
0 votes
1 answer
395 views

Still can't connect from my grafana docker container to prometheus

...after having followed a bunch of tutorials, e.g. https://medium.com/@TimvanBaarsen/how-to-connect-to-the-docker-host-from-inside-a-docker-container-112b4c71bc66 I started the grafana container with:...
user avatar
0 votes
0 answers
16 views

GKE collector metrics missed some metrics

[enter image description here][1]On the cluster in GKE, we have a CronJob deployed, which sends prometheus metrics to the handler. A collector is launched to it using PodMonitor. I understand that it ...
user avatar
0 votes
0 answers
60 views

Autodiscovery in prometheus

This might be a newbie project so please be patient till the end of the post. I'm developing a demo data centre and have commissioned a couple of machines to act as nodes. My DC is defined as follows: ...
user avatar
0 votes
0 answers
34 views

CentOS - listing size of top-level directories in a 50TB NTFS volume is way too slow

What I want to achieve is get the information about how big are the top-level folders in a directory (which is a NTFS volume) on a CentOS7 server. This information is placed in a Prometheus file, ...
user avatar
0 votes
0 answers
103 views

microk8s: pod resource usage metrics not available from all nodes

I am running microk8s v1.22/stable on a Linux cluster with 11 nodes. I have enabled the metrics-server plugin and installed Prometheus via the Helm chart with nodeExporter and kubeStateMetrics enabled....
user avatar
  • 101
0 votes
1 answer
129 views

./node_exporter: cannot execute binary file

I am using RHEL7.9 in my Virtual Box. I installed the binary file of node_exporter as explained in the official documentation. I tried to run node_exporter using the following command: ./node_exporter ...
user avatar
3 votes
1 answer
436 views

Kubernetes Nginx Ingress Controller Metrics

I've tried to find a documentation about the metrics exposed by the NGINX ingress controller in Kubernetes but so far I haven't found any reliable source about the metrics and what they mean. For ...
user avatar
0 votes
0 answers
22 views

Rules, rules.d, files_sd for prometheus while installing in RHEL

I was trying to install prometheus in RHEL7 by following the tutorial of this link. I was on this step which showed a demo example of the code as following: Code Here, what would I write on the place ...
user avatar
0 votes
0 answers
21 views

Dependencies to install Prometheus 2.32.1 in RHEL 7.9

I am trying to install prometheus-2.32.1 in RHEL 7.9 in my virtualbox. But I am failing since it says that yum repository does not have the necessary repos. Can anybody tell me what are the ...
user avatar
1 vote
0 answers
83 views

getting prometheus metrics from pdns

Running pdns v4.7.0-0.alpha0.master. Have enabled web server and can see ports 8081, 8081 and 8082 (from localhost login). When I try to get /metrics from there (via curl) I get either "...
user avatar
  • 2,297
0 votes
1 answer
30 views

Is there a daemon that can poll Prometheus targets and relay to Graphite?

I am running a Graphite based collection, and have a couple of Prometheus only apps that I would like to be able to monitor through that. Does anyone know of a proxy-like piece of software that can ...
user avatar
0 votes
0 answers
59 views

How to monitor docker container network usage with Prometheus?

I would like to monitor container level metrics for network usage, these are displayed by docker stats but not exposed by the native Docker Prometheus exporter (https://docs.docker.com/config/daemon/...
user avatar
0 votes
0 answers
85 views

Prometheus Alertmanager - Generating alerts

When alerts are based on metric data, like CPU or memory utilization, Prometheus is the obvious tool for pushing alerts into Alertmanager. There are other examples where the required alert is based ...
user avatar
0 votes
1 answer
60 views

Prometheus Alert Manager - CPU high not alerting

I configured prometheus alert manager, but he is not alerting when the CPU of one of my server goes to 99% of usage. This is the alert : - alert: HostHighCpuLoad expr: avg(irate(...
user avatar
0 votes
1 answer
42 views

Better way at scale to pull image uris from all pod specs on k8s cluster

Team, I see this [list-all-running-container-image]https://kubernetes.io/docs/tasks/access-application-cluster/list-all-running-container-images/. however, I cannot bank operations on this because it ...
user avatar
  • 109
0 votes
1 answer
1k views

Wipe all stored prometheus metrics?

I have install prometheus (v2.1.0 from Ubuntu 18.04's apt repository), and I have played around with adding hosts & metrics and all that for about a week. I want to “start fresh” and delete all ...
user avatar
0 votes
1 answer
35 views

Query apache for how many responses (of each status code) it has served out for better monitoring & graphing?

I have several apache web servers installed from standard Ubuntu 18.04/20.04/etc. I want to monitor how many requests they are handeling. I am now looking into prometheus & grafana after 15+ years ...
user avatar
0 votes
0 answers
56 views

Unable to scrape kublet api from prometheus

I am setting up prometheus to scrape kubernetes cluster. I am trying to use "role: node" with kubernetes_sd_config to monitor one of the K8s cluster. I created certificate ashishcert.pem for ...
user avatar
0 votes
0 answers
67 views

Prometheus alert rule test fails with long intervals

I've got an alert rule: groups: - name: somename rules: - alert: CertificateExpiry expr: ...
user avatar
  • 878
0 votes
0 answers
245 views

Grafana plugins not working

I have a grafana server (6.3.5) and I'd like to add the Flowcharting plugin. I've followed the installation instructions in the official documentation and did as follow : sudo grafana-cli plugins ...
user avatar
  • 91
0 votes
1 answer
550 views

Can I tail the log of ONE container in a docker-compose group?

I'm still struggling to understand why my ipmi_exporter container will not run in the same "pod" as my Prometheus, Grafana, snmp_exporter and alertmanager. When I bring up the "pod,&...
user avatar
  • 902
1 vote
1 answer
400 views

Why does my prometheus dashboard only show 2 weeks of data when I have longer term data available?

I have Prometheus configured with Thanos. I'm using the Node Exporter Full dashboard to view data from the node exporter. Regardless of what time range I select (e.g., now-20d), the dashboard only ...
user avatar
  • 40.4k
1 vote
1 answer
404 views

Why K8S statefulsets volumeClaimTemplates status is pending , but the pod, pvc, pv are all fine?

I use nfs-subdir-external-provisioner as automatic pv provisioner for my promethues (by promethues-operator ) I have created our sts,pod,pvc,pv successfully and everything looks fine. But if I use ...
user avatar
  • 13
0 votes
0 answers
205 views

How can I get nginx to add a newline to a response from an upstream server?

I have a service that exposes a Prometheus statistics endpoint. I'm using Telegraf's input.prometheus to relay these stats. Unfortunately this particular service doesn't terminate the request with a ...
user avatar
  • 111
0 votes
0 answers
23 views

Promtheus Increasing metric show increments by hours in a day

I have a host that prints and I want to show how many prints are done by the hour for a day and average for the past x days. The current value is 1000, I expect this to increment about 100 per day. ...
user avatar
  • 1
2 votes
2 answers
2k views

Netdata, prometheus and grafana - why recommended in this constellation?

Currently I do setup monitoring-service for our Server. I've found some articles, which recommend to use: Netdata, prometheus and grafana So I wonder what are the benefits to use all three of them. ...
user avatar
  • 133
1 vote
2 answers
898 views

Prometheus not connected to alert manager in GKE

I installed kube-prometheus-stack 15.3.1 into a GKE cluster using helm (in "monitoring" namespace). I used the values.yaml to open up ingresses on some of the components and to add SMTP info ...
user avatar
-1 votes
1 answer
190 views

How to track down this spike in Disk IO every 5 minutes on Ubuntu Server / microk8s

I've set up an Ubuntu Server with microk8s, with the dns, dashboard and prometheus addons. It's running some Cardano nodes. On the (built-in) Grafana dashboard "Default / Nodes" I see spikes ...
user avatar
-1 votes
1 answer
294 views

Blackbox-exporter modules header

How to provide headers in modules in blackbox.yml. module: prober: http http: headers: Authorization : "----" This returns in error parsing config file:yaml: unmarshal errors
user avatar
  • 9
1 vote
0 answers
122 views

How are Prometheus Snapshots stored on disk?

I am trying to understand how to scale my Prometheus and look at the storage mechanic for this. Lets assume the following: Prometheus data storage Directory: 20GB in size Snapshots amount: 3 Snapshot ...
user avatar
  • 111
0 votes
0 answers
1k views

nginx: [emerg] unknown directive "lua_shared_dict" in /etc/nginx/nginx.conf:41

Code from my nginx.conf file: user www-data; worker_processes auto; pid /run/nginx.pid; # remote the escape char if you are going to use this config include /etc/nginx/modules-enabled/\*.conf; events ...
user avatar
0 votes
1 answer
430 views

Prometheus Monitor Host Network Traffic From Within Docker Container

Is it possible / intended to use a prom/node-exporter container to monitor the network traffic of the docker host? In my setup, all the host interfaces show up in node_network_iface_id but ...
user avatar
2 votes
1 answer
118 views

Spikes on External Metric scales HPA when it shouldn't

I have a metric that I’m using for a HPA, the problem is that the metric has spikes and to avoid so, I’m using an average over time in a record rule on prometheus, but to export it to stackdriver(on ...
user avatar
2 votes
2 answers
9k views

How do I configure the log level of Prometheus' Alertmanager?

I've set up Prometheus' Alertmanager to, well, manage alerts from Prometheus. I've got the alerts coming in from Prometheus to Alertmanager, but there the story ends. The alerts never get sent on by ...
user avatar
  • 549
2 votes
1 answer
3k views

Prometheus Alerting on NodeClockNotSynchronising for VMs

I'm trying to determine why this Alert (NodeClockNotSynchronising) is firing for a handful of VMs I've provisioned. (Not all just a few, which is strange) According to the metrics that are exported, I'...
user avatar
  • 123