Questions tagged [prometheus]

The Prometheus monitoring system.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

Prometheus absent function for 800 nodes

I have specific requirement from the client that prometheus should generate alert in case any mount point on the server goes missing. For Eg: If server has 3 mount points like /data1 /NFS1 /NFS2 and ...
-1
votes
1answer
23 views

Blackbox-exporter modules header

How to provide headers in modules in blackbox.yml. module: prober: http http: headers: Authorization : "----" This returns in error parsing config file:yaml: unmarshal errors
0
votes
0answers
29 views

Install or rebuild package from Ubunntu in Centos 7

I'm new to building packages in repository so please excuse me if I'm asking something basic or commonly known. I need to build a package from source : https://github.com/kumina/postfix_exporter which ...
1
vote
0answers
25 views

How are Prometheus Snapshots stored on disk?

I am trying to understand how to scale my Prometheus and look at the storage mechanic for this. Lets assume the following: Prometheus data storage Directory: 20GB in size Snapshots amount: 3 Snapshot ...
0
votes
0answers
224 views

nginx: [emerg] unknown directive “lua_shared_dict” in /etc/nginx/nginx.conf:41

Code from my nginx.conf file: user www-data; worker_processes auto; pid /run/nginx.pid; # remote the escape char if you are going to use this config include /etc/nginx/modules-enabled/\*.conf; events ...
0
votes
0answers
65 views

Prometheus Monitor Host Network Traffic From Within Docker Container

Is it possible / intended to use a prom/node-exporter container to monitor the network traffic of the docker host? In my setup, all the host interfaces show up in node_network_iface_id but ...
3
votes
1answer
81 views

Spikes on External Metric scales HPA when it shouldn't

I have a metric that I’m using for a HPA, the problem is that the metric has spikes and to avoid so, I’m using an average over time in a record rule on prometheus, but to export it to stackdriver(on ...
0
votes
0answers
106 views

Prometheus Helm - alert manager doesn't receive any alerts and no emails either

I tried to kill the node exporter and then expected to get emails. I've configured it to send email even if its down for 1 seconds but nothing happens. I don't see the alerts in alertmanager either.. ...
1
vote
1answer
1k views

How do I configure the log level of Prometheus' Alertmanager?

I've set up Prometheus' Alertmanager to, well, manage alerts from Prometheus. I've got the alerts coming in from Prometheus to Alertmanager, but there the story ends. The alerts never get sent on by ...
1
vote
1answer
443 views

Prometheus Alerting on NodeClockNotSynchronising for VMs

I'm trying to determine why this Alert (NodeClockNotSynchronising) is firing for a handful of VMs I've provisioned. (Not all just a few, which is strange) According to the metrics that are exported, I'...
0
votes
0answers
56 views

Prometheus scraping issue with kafka-exporter

I am using kafka-exporter to be as a scraping endpoint for Kafka metrics. After installation of kafka exporter Prometheus does not collect and display any Kafka metric. Some extra information: i am ...
0
votes
1answer
38 views

realtime monitoring of linux server network

We have a few Linux server which act as Edge network, We want realtime (or near-realtime) information on network bandwidth, top IP addresses and network bandwidth per them and other network parameters ...
0
votes
0answers
29 views

Get specified value in prometheus target labels?

Brief: I have an kubernetes cluster container a default, monitoring and multiple shoot-******** namespace. (******** is random) I setup prometheus in kubernetes monitoring namespace, and kube-state-...
1
vote
1answer
289 views

Prometheus auto scrape metrics from multiple kube-state-metrics in kubernetes?

I want to use a kubernetes(cluster-0)with multiple kube-state-metrics to monitor multiple other kubernetes cluster(cluster-1,2,3,4) In the (cluster-0), I split into multiple namespaces like this: ...
0
votes
0answers
41 views

how to set a Prometheus alert if a specific pod is not in running the state?

how to set a Prometheus alert if a specific pod is not in running the state? basically, this pod has a static prefix podname i.e sam-xxxxxxxxx and the suffix will be changed when it gets recreated for ...
0
votes
0answers
27 views

Pause Prometheus during Off-time

We have servers that shutdown during certain off-times to save cost. For example Server1 shuts down between 23:00 and 7:00 - during that time the server should not be checked and no alerts should be ...
0
votes
0answers
21 views

Sending telegraf socket_listener metrics to Prometheus' AlertManager

I'm deploying a monitoring solution that is bit of an hybrid solution (not my choice, it's a corporate hard-requirement): Grafana for visualisation Telegraf agents push metrics to InfluxDB Prometheus ...
0
votes
0answers
213 views

How to get heroku metrics into prometheus?

We have a diverse infrastructure. Historically, frontends got deployed to heroku, while backends run on other cloud infrastructure. We have a prometheus installation for monitoring the backends set up....
0
votes
0answers
6 views

gerrit statistics - parsing message flow - script

I want to measure How long does it take from developer Uploaded patch set, to message build ok from Jenkins, as seen in gerrit message flow. I would like to parse message flow - what is the best ...
0
votes
1answer
1k views

Prometheus: Get CPU Usage % for one specific process from windows_exporter

I am trying to develop one query to show the CPU Usage(%) for one specific process in one windows server. After digging into the metrics of windows exporter, I found Metric: process....
0
votes
1answer
445 views

Prometheus - Use case of service discovery by role endpoints and role pod in Kubernetes

While reading Prometheus Configuration documentation and some sample scrape configurations, I found some kubernetes_sd_configs with role service & role endpoints & role pod - job_name: kube-...
0
votes
1answer
152 views

Prometheus Alert Manager error “component=cluster err=”couldn't deduce an advertise address“ and ”unable to initialize gossip mesh"

I'm new to Prometheus and I'm trying to install Alert Manager module. Using Prometheus 2.19.2, Node exporter 1.0.1 and Alert Manager 0.21.0 on Debian 10. Configuration files: Prometheus: alerting: ...
0
votes
0answers
34 views

Is it possible to display successful queries of an arbitrary endpoint in prometheus

We have nodes which offer a status endpoint as a plain simple GET: GET /status It returns just simple 200 if all is ok, otherwise an error code. I would like to display this in Grafana via Prometheus, ...
0
votes
2answers
98 views

cAdvisor - sync between nodes

I have a docker swarm running our business stack defined in a docker-compose.yml on two servers (nodes). The docker-compose has defined cAdvisor starting on each of the two nodes like that: cadvisor:...
0
votes
0answers
28 views

Consul handling dynamic IP address

If suppose I have a node-exporter for my project running on aws instance its IP:port is:- "10.0.0.1:9100". I used the agent service register to add it to the consul. Then used consul_sd to ...
0
votes
0answers
146 views

Kubernetes: Prometheus operator (helm) not auto-scrapes metric from VerneMQ Service Monitor

I'm using Helm to install prometheus-operator in my kubernetes cluster, everything work fine and I can get the metrics root@k8s-master:~/vernemq# kubectl get pod NAME ...
2
votes
1answer
133 views

Pod using Vernemq helm package cannot start

I'm using helm to install vernemq on my kubernetes cluster The problems is it can't start, I accepted the EULA Here is the log: 02:31:56.552 [error] CRASH REPORT Process <0.195.0> with 0 ...
1
vote
2answers
241 views

Breaking down one prometheus.yml file?

I am using Prometheus for our monitoring and I have a lot of configs (our prometheus.yml main config file is 8000+ lines long). I would like to divide this out into logical groupings so that it ...
1
vote
2answers
1k views

What does it mean to have more than one instance of Prometheus in Kubernetes

Suppose I'm using a volume to persist my Prometheus data, I wonder if I can have more than one instance of it running to have high availability. I believe only one instance of Prometheus must be in ...
0
votes
2answers
283 views

Missing metrics for “kubelet_volume_*” in Prometheus

I setup latest https://github.com/coreos/kube-prometheus/ in an AWS EKS cluster in which I'm using the Amazon EBS CSI driver for persistent volume claims, but I don't see any "kubelet_volume_*" ...
1
vote
0answers
86 views

kube-state-metrics doesn't show hardware utilization

I installed this yamls https://github.com/kubernetes/kube-state-metrics/tree/master/examples/standard I can see CPU per pod utilization only from system pods such as: calico-node, coredns, ...
1
vote
1answer
112 views

Gitlab prometheus error after upgrade with apt-get upgrade

after upgrading my gitlab from 10.9.2 to version 10.10 on my Ubuntu 18.04 it was not possible for the apt-get upgrade process to end successfully. It ended with following message: Ensuring Prometheus ...
0
votes
0answers
102 views

How to expose redmine metrics to prometheus?

I have deployed redmine over kubernetes and a monitoring stack consisting prometheus, grafana and alertmanager. I have to perform application monitoring through prometheus for which I need to expose ...
2
votes
1answer
47 views

How to trigger alerts in Prometheus when specific users login to OpenShift or Kubernetes?

Using either kube_state_metrics or anything else I'd like to fire alerts in Prometheus AlertManager when a specific user logs in to the cluster, ie. kubeadmin or bob-smith. Or in other words: where in ...
3
votes
1answer
321 views

Prometheus node_exporter metrics - huge spikes in Grafana

Our setup includes: a few Debian 9.12 nodes with Prometheus node_exporter v. 0.18.1 installed as service Prometheus server v. 2.14.0 (on Windows Server 2016) scraping the metrics from nodes Grafana ...
1
vote
0answers
474 views

Monitoring SLA/SLO/SLI using Prometheus

I have done much research about monitoring SLI metrics with Prometheus. I have found only how to monitor a cluster using Kubernetes. I'm hoping to find a response here for simple monitoring. I also ...
0
votes
1answer
22 views

How to run containerized prometheus, monitor host and sent to grafana

I have two servers with Ubuntu 18.x. On one I have installed the official dockerized Grafana. What I want to do now is installing a dockerized version of Prometheus on the second server and get the ...
1
vote
1answer
2k views

Promethius, group_left, and “on” vs “ignoring”

In Issue #2204, one of the Prometheus developers says: ...in principle you should be favouring ignoring over on to produce generic shareable rules... I'm confused how the use of ignoring would ...
2
votes
1answer
388 views

Do prometheus metrics have some sort of “freshness” lifetime?

I'm scraping some metrics (openstack cinder volume sizes) every 15 minutes, and the results produce a discontinuous graph, like this: (That's the result of the simple query cinder_volume_size_gb). ...
1
vote
1answer
192 views

Filtering interface results from prometheus snmp-exporter by interface name

I'm collecting interface statistics from some switches using the following configuration: modules: if_mib: walk: - ifTable auth: community: monitor The data returned from the ...
0
votes
1answer
93 views

Aws coudwatch metrics convergence

We are in the process of setting up aws cloudwatch exporter(https://github.com/prometheus/cloudwatch_exporter) to export cloudwatch metrics to our prometheus server. When reading through the ...
1
vote
1answer
511 views

Monitor Gitea with Prometheus

I have deployed Gitea. How can I want to monitor it with Prometheus ? I have tried to scape the /metrics but I get no metrics (HTTP/404)
2
votes
1answer
925 views

Prometheus: scrape interval is 1m, but resolution is still 15s

tl;dr: My scrape interval is 1m, yet I have a 15s resolution. Why? My prometheus configuration includes a job to scrape kong metrics: - job_name: kong_blue honor_timestamps: true scrape_interval:...
1
vote
0answers
1k views

Convert json file to prometheus metrics

I have nginx json access log like this, a file that append sample like this to end of file. { "time": "2019-11-27T08:23:39+00:00", "msec": 1574843019.787, "ip": "91.133.248.69", "agent_name": ...
1
vote
0answers
23 views

Prometheus: how to stop alerting on a dead system

I've got my backup server reporting the age of the last successful backup to Prometheus, and I alert when that gets too old. I've just permanently shut down a server, and of course Prometheus is now ...
0
votes
0answers
223 views

Some Prometheus alerts mislabelled

We've been using Prometheus with fairly solid success so far. However, we've recently noticed that one of the sets of alerts we've set up ends up mislabelled: It shows up as a different environment ...
2
votes
2answers
3k views

Smoothing Grafana graph

I'm setting up montinoring on my company infrastructure. I installed Grafana with the Prometheus Datasource. I built my dashboards and have rather "Spiky" graphs. I was wondering if there was any way ...
0
votes
1answer
75 views

Monitoring EKS Kubernetes LoadBalancer service Type

I have created few services in Kubernetes with type: LoadBalancer. Platform: EKS. Is there a way to get number of 4xx or 5xx errors from this LoadBalancer? I have tried the following: Prometheus -...
2
votes
2answers
3k views

Disk usage metrics missing from Prometheus node exporter

I used Prometheus and node exporter a while ago and had access to node_filesystem_* metrics to monitor disk usage but I've recently fired it up on some other servers (Ubuntu Linux) and those metrics ...
1
vote
1answer
2k views

How to monitor ping packet loss via prometheus exporters?

I'm already trying to use this exporters for simple icmp probes: https://github.com/prometheus/blackbox_exporter It gives only 3 icmp latency metrics https://github.com/czerwonk/atlas_exporter ...