Questions tagged [prometheus]
The Prometheus monitoring system.
139
questions
0
votes
0
answers
10
views
Correct way to setup a multinode LGTM stack
I have a 4 nodes clusters:
MonitoringCenter, hosting
a Grafana, connected to all Prometheus and Loki instances + local AlertManager
a Prometheus scraping local NodeExporter/AlertManager/Loki/...
- 199
0
votes
0
answers
10
views
How to add configuration for fluent-plugin-prometheus in Fluentd deployed via Fleet in Rancher?
I'm using Rancher to manage my Kubernetes cluster and have added a logging system (cattle-logging-system) via Fleet. I now need to add monitoring for Fluentd using the fluent-plugin-prometheus.
Here's ...
0
votes
0
answers
20
views
Thanos Receiver not deleting old data in Persistent Volume (PV) after retention is exceeded
I have set up Kube Prometheus Stack with Thanos on my Kubernetes cluster, and I'm using the Thanos Receiver instead of the sidecar approach. I have also configured the Thanos Compactor and Minio for ...
0
votes
0
answers
15
views
Spike in Cadvisor container_network_receive_bytes_total Metric in a Kubernetes Cluster
Summary:
I'm using Cadvisor with Prometheus in multiple Kubernetes (k8s) clusters to monitor network traffic usage. I utilize the container_network_receive_bytes_total metric in a query to calculate ...
0
votes
0
answers
37
views
Prometheus Server Pod Suddenly Crashed (unexpected fault address 0x7f911b1795d4)
With traffic Prometheus server pod getting restarting with below error stack. This is happend when the live traffic of the system. But could not be able to reproduced with the load testing.
Grafana ...
1
vote
1
answer
36
views
How to safely update an existing Kubernetes server without original configurations
I was handed a Kubernetes cluster with no config files, and was not setup with helm. The author said they just created everything from the cmd line. It is a small/new cluster for a single API server ...
- 13
0
votes
0
answers
16
views
Restrict access to a Prometheus server in AKS can only be achieved with nginx-ingress?
Prometheus server with its respective Loadbalancer in AKS.
I wanted to secure the access to /metrics through network rules...but it doesn't work. I can still acess to the endpoint with any device.
...
- 1
0
votes
0
answers
59
views
How can I know request waiting time in Nginx Ingress Controller?
We use Kubernetes with Nginx Ingress Controller to run our platform with various backend services. We also use New Relic (& Prometheus, Grafana) for our Observability dashboards & alerts. ...
- 101
0
votes
1
answer
64
views
prometheus-operator when configuring alertmanager config for PagerDuty
Have next issue:
When try to set up alertmanager configuration over CRD, get wrong configuration on pod.
Problem look like:
- routing_key: |
*****
routing_key parameter after all CD use ...
0
votes
1
answer
104
views
promtool tsdb create-blocks-from openmetrics doesnt show
I have a file like
# HELP process_cpu_user_seconds_total Total user CPU time spent in seconds.
# TYPE process_cpu_user_seconds_total counter
process_cpu_user_seconds_total{NODE_APP_INSTANCE="...
- 101
0
votes
0
answers
54
views
how to configure a prometheus installation of openshift to include the labels of node role in the exporter?
How to configure a prometheus installation of openshift to include the labels of node role in the exporter?
We want to alert based on cpu requests only on worker nodes
namespace_cpu:...
0
votes
0
answers
81
views
kube-prometheus-stack redundancy across multiple clusters
I currently use kube-prometheus-stack to monitor several kubernetes clusters. Each cluster has its own deployment of the kube-prometheus-stack, however, there is currently only one cluster (a) that ...
- 31
1
vote
0
answers
63
views
my prometheus dashboard shows 0 on node_export instance
I have installed prometheus on a server and node_exporter on another server
I configured my prometheus.yml to interact with the node exporter
The connection works fine but the up query shows 0 which ...
- 113
0
votes
0
answers
48
views
How to export Prometheus metrics from GCE into stack driver
We currently have some software that contains Prometheus data on a GCE. The said GCE already has the legacy logging agent using fluentD running, this is currently incompatible with the google ops ...
- 1
0
votes
0
answers
156
views
Persist Prometheus data from a Docker Container
I'm running Prometheus and Grafana from docker-compose and basically it works. But I don't get my prometheus data to persist on my host machine.
As soon as I switch my volume settings from
---
version:...
0
votes
0
answers
349
views
How to scrape Prometheus secured with OAuth2-proxy and Keycloak
I have 2 prometheuses, both are with forward-auth via the oauth2-proxy, which have the same client credentials in a single keycloak.
I would like one prometheus to federate the other one. This is my ...
- 373
1
vote
2
answers
63
views
Monitor Azure Kubernetes Service (AKS) with Prometheus servers running independently
Can I deploy Prometheus to each Azure Kubernetes Service (AKS), and monitor each Prometheus from a Prometheus created by a virtual machine outside the AKS? In other words, I want to connect the ...
- 33
2
votes
2
answers
662
views
Debugging Prometheus OOMkilled despite 6Gi limits
I'm at the end of my patience with a prometheus setup leveraging kube-prometheus-stack 44.3.0 (latest being 45).
I have two environments, staging and prod. In staging, my prometheus runs smoothly. In ...
- 131
0
votes
0
answers
37
views
How to avoid duplicate entries in DNS Bind cache db file?
While running command "rndc dumpdb -cache" it has huge amount of data some duplicate entries like one domain has multiple entries with same target but different TTL, how to reduce like ...
- 1
0
votes
1
answer
193
views
How to remove labels in alert template?
I found a template for Prometheus Alert Manager for alerts, but I receive many unnecessary labels in notification. How to remove labels in template? This code is responsible for labels (look below). I ...
- 131
0
votes
0
answers
371
views
How to remove information about old firing alerts in notification?
When 1 alert is firing, AlertManager sends a notification, then 2nd alert is firing or resolved, AM sends a notification with information about 2 alerts. How to remove information about 1st alert in ...
- 131
0
votes
1
answer
584
views
Monitoring of MySQL replication and alerting when replication is not working
I have a question about monitoring of MySQL replication and in particular about alerting using Grafana/Prometheus.
We have 2 MySQL(MariaDB to be exact) instances for which replication is configured: ...
- 11
0
votes
0
answers
77
views
Nginx `ngx_http_stub_status_module` with multiple sites on the same server (VM)
Server is a VM running Debian 11 with nginx-extras package.
I have the ngx_http_stub_status_module setup in /etc/nginx/conf.d/stub_status.conf as shown below.
server {
listen 127.0.0.1:8080;
...
- 101
1
vote
1
answer
377
views
Grafana graph value as a percentage of the initial value
I am new to grafana & prometheus. I want to graph the change in a value (here disk space used), but as a percentage of the initial value, which one commonly sees for inflation charts.
For example, ...
- 31.5k
0
votes
1
answer
391
views
Prometheus expose any metrics on itself? (e.g. disk space it uses)
Does Prometheus expose any metrics on itself? i.e. how much disk space it's using etc.
I want to start fine tuning our prometheus server, so need to monitor what's currently there. I want to be able ...
- 31.5k
0
votes
0
answers
83
views
Agent prometheus does not send metrics
I am trying to configure a Prometheus agent with the 'remote_write' function. The Prometheus agent configuration file has these parameters.
global: scrape_interval: 30s external_labels:
...
- 1
0
votes
1
answer
1k
views
kube-api server high cpu
I want to know how I can check why one of my ctrl node and kubernetes consumes more cpu than the others.
I have a cluster with 3 ctrl nodes and 4 worker nodes.
I have an nginx load balancer with the ...
PeterWegner90
0
votes
0
answers
57
views
Accessing local host Dashboards (GCP VM) from my Laptop
I have 2 VM instances on Google Cloud Platform.
1st VM is used as a Kubernetes Master Node and 2nd as Worker Node.
I have Prometheus and Grafana Deployed on the cluster. I want to access the ...
- 13
0
votes
1
answer
1k
views
Kubernetes upgrade from 1.21 to 1.22 caused Prometheus to fail
We recently upgraded Kubernetes 1.21 to 1.22 version on aws eks. The upgrade was successful. However, the associated prometheus deployments fails with error
$ kubectl -n monitoring logs prometheus-...
- 103
0
votes
0
answers
24
views
Prometheus Metrics - Multiple boxes in one file
My existing custom exporter for Prometheus generates:
this_metric_one 10.3
this_metric_two 11.4
Currently this is running on two boxes, Box_A and Box_B. INSTEAD, I want to have Box_C query both (...
- 101
0
votes
0
answers
412
views
Alertmanager does not send resolved email
I have configured an alertmanager and it sends alerts properly But it does not send resolved email. I have tried different amount of time for resovled timeout but it did not work
here is my ...
0
votes
0
answers
50
views
My mysql Exportir not appears on my Prometheus localhost
So I have 2 VMs in my virtualbox, there are Monitor and Target. So, in the Monitor VM, I install Grafana as monitoring and in Target VM, I install Prometheus as target monitoring.
For now I've ...
0
votes
0
answers
102
views
Prometheus scraping halted for last 20 minutes
My prometheus dashboard has values missing for the last 20 minutes, no matter when I query (container_cpu_usage_seconds_total) it. What could be the reason behind this?
Screenshot of dashboard
- 1
0
votes
0
answers
47
views
What is the best way to set up Prometheus + Alertmanager with a HA
I have a monitoring solution that uses Prometheus as a scraper and a data storage, Grafana as a visualiser and Alertmanager as an alerting tool. This all is running on a single server.
However, there'...
- 413
1
vote
1
answer
523
views
Ceph alertmanager config
I've installed ceph using cephadm
Including monitoring stack prometheus, alertmanager, node-exporter
Currently I'm trying to add a telegram receiver (Telegram is supported from v0.24.0, so I've ...
- 63
0
votes
1
answer
624
views
promtool tsdb create-blocks-from openmetrics doesn't show any data in Prometheus
I'm trying to backfill Prometheus with old data. My input data is:
$ cat /etc/prometheus/backfill.txt
# HELP mymetric My metric.
# TYPE mymetric counter
mymetric{code="200",service="...
- 131
0
votes
1
answer
2k
views
Prometheus: Add insecure_skip_verify via annotation or scrape config adaption for kubernetes pods
I`ve running a kubernetes cluster with a deployment of some pods. One pod provides metrics on a https secured endpoint. The problem is, that this pod create and use his own self-signed certificate and ...
- 347
0
votes
0
answers
96
views
Prometheus query to identify active series if a datapoint is sent in the last 15-30mins
I am trying to identify my Prometheus active series if a datapoint is sent in the last 15-30mins.
As I understand it, this query prometheus_tsdb_head_series gives me active series if a datapoint is ...
- 1
1
vote
2
answers
3k
views
Grafana pod is not running, how to fix that?
I have deployed grafana in eks using the steps provided in this link
After deployment of grafana, the pod is not in running state.
kubectl get po -n grafana
NAME READY STATUS ...
- 155
-1
votes
2
answers
151
views
RHEL + can we improve disks performance by tuning kernel parameters?
we have Hadoop cluster and we are collection metrics collection data in order to investigate slowness behavior on spark applications
after long investigation on our Hadoop cluster
we noticed from ...
- 559
2
votes
0
answers
243
views
Does Prometheus efficiently compress series of identical samples?
Does Prometheus efficiently store and de-duplicate long repeats of identical sample values for a metric?
I'm working with a system where some of the inputs change at a low frequency - say every 30 ...
- 11.1k
0
votes
1
answer
4k
views
prometheus field remote_write not found in type config.plain
I need help using prometheus remote_write and remote_read to/from influxdb.
I got error below:
# promtool check config /etc/prometheus/prometheus.yml
Checking /etc/prometheus/prometheus.yml
FAILED: ...
- 1
1
vote
0
answers
3k
views
Hostname instead of IPs for panel legends in Grafana with Prometheus metrics
I want to monitor different servers with Prometheus fed by exporters like node-exporter or postgres-exporter. The metrices should be presented in Grafana as dashboards and alerts.
The exporters will ...
- 113
0
votes
1
answer
921
views
Disabling email for Prometheus Alertmanager based on severity
We are monitoring a HPC computing cluster using a combination of Prometheus, Alertmanager and Grafana. On our machines, things like SWAP memory filling up to essentially the limit happen frequently, ...
- 101
1
vote
1
answer
4k
views
Alertmanager telegram config chat_id and cannot unmarshal errror
I am trying to configure alertmanager to send alerts to my telegram group. Following the configuration I have:
global:
resolve_timeout: 5m
route:
group_by:
- job
group_interval: 5m
...
- 21
0
votes
1
answer
999
views
Change retention time on prometheus
So I am running Prometheus on an Ubuntu server and started running it a while ago by just running
sudo ./prometheus
However, I wanted to append
--storage.tsdb.retention.time=1y
to the end of this ...
0
votes
1
answer
921
views
webhook MS Teams integration with Prometheus - request failed
I'm struggling with Microsoft Teams/Prometheus integration on K8s cluster.
I used helm to start all components.
I have correctly working Prometheus and Alertmanager. It seems that all works fine. ...
- 31
0
votes
1
answer
345
views
Nginx reverse proxy when port is already in use?
i am trying to close port with basic authentication (for pushgateway of prometheus), so not a big specialist in nginx, so could someone please give me and advice where i am is wrong?
I have 9091 port, ...
- 103
0
votes
1
answer
2k
views
Debug Prometheus Blackbox Exporter http_2xx probs
We are using the Prometheus Blackbox Exporter (blackbox_exporter, version 0.19.0) to check HTTP endpoints.
Now we need to implement a check with the fail_if_body_matches_regexp.
The check should ...
- 121
1
vote
1
answer
2k
views
Still can't connect from my grafana docker container to prometheus
...after having followed a bunch of tutorials, e.g.
https://medium.com/@TimvanBaarsen/how-to-connect-to-the-docker-host-from-inside-a-docker-container-112b4c71bc66
I started the grafana container with:...
- 499