18 votes

How to find out the number of time series stored in Prometheus LevelDB

How about count({__name__=~".+"})? It does return number of time series in the database. I compared with amount of metrics currently exposed by each target by manual scrapping, and it matches it +/- ...
  • 457
15 votes
Accepted

Prometheus: no metric node_cpu

The 0.16.0 version of the node exporter improved many key metrics to follow naming guidelines that arose since it was created, so node_cpu_seconds_total is the new name. Many dashboards etc. are still ...
  • 3,904
10 votes
Accepted

How to find out the number of time series stored in Prometheus LevelDB

prometheus_tsdb_head_series Just because I always forget and have to Google this, and this question is at the top. As per this answer there will likely be a variance between ...
  • 2,340
9 votes
Accepted

How to split out Prometheus config files?

The Prometheus config file (and other config files in the ecosystem) explicitly do not support any form of templating. Instead this is left up to your configuration management system to handle. ...
  • 3,904
9 votes

Prometheus Alert Rule for Absent Discovered Target

Or do I need to hard code rules for each service/node/pod explicitly, even though it was auto discovered? Yes, you need a rule for every individual thing to you to alert on being missing as ...
  • 3,904
6 votes

Prometheus Alert Rule for Absent Discovered Target

We've been solving something similar. Our setup: when some service starts somewhere, some metrics appear with a non-zero value. Then, if any of those metrics go missing, we want an alert. In our case,...
  • 387
6 votes

Netdata, prometheus and grafana - why recommended in this constellation?

In summary, Grafana will display metrics collected by Prometheus from netdata. There's no benefit to rolling out a stack like that except that Prometheus has scaling capabilities that has made it ...
  • 1,998
5 votes

How to find out the number of time series stored in Prometheus LevelDB

https://prometheusui.com:9090/status ---> Head Stats --> this will give you the entire status.
  • 111
4 votes

How to find out the number of time series stored in Prometheus LevelDB

After some more research and thanks to the comments of David.B i found a solution that 'works for me' ™ To find the number of time series stored by Prometheus i use this command in the storage....
4 votes

How to monitor REST API latency with Prometheus and Grafana

You could use my tomcat_exporter to do this by configuring the filter for the response time metrics. Alternatively you could implement a similar filter by yourself.
  • 41
3 votes
Accepted

Kubernetes: horizontal auto-scaling based on metrics in another namespace

Ah, I've figured it out. Here's the part of the prometheus-adapter configuration I needed: rules: - seriesQuery: '{__name__=~"^nginx_ingress_.*",namespace!=""}' seriesFilters: [] ...
3 votes

Monitor Gitea with Prometheus

Gitea can export Prometheus metrics (since ~ 1.7.1) , but that's disabled by default. In the configuration file conf/app.ini, you should add: [metrics] ENABLED=true The metrics are exposed on the ...
3 votes

What does it mean to have more than one instance of Prometheus in Kubernetes

You can also run two instances/replicas of Prometheus in parallel, both scraping targets and saving the data independently, then have another service like Thanos Query in front of them to query both ...
3 votes
Accepted

How do I configure the log level of Prometheus' Alertmanager?

The answer is that its not possible to set Prometheus' Alertmanagers log level to debug through the config file, it is only possible through commandline arguments. Do not ask me why, I'm sure they had ...
  • 569
3 votes

Netdata, prometheus and grafana - why recommended in this constellation?

I would suggest, that if you are just starting out, a combination of the Netdata Agent + Netdata Cloud has you covered. You install the Netdata Agent on every computer that you wish to monitor and you ...
3 votes
Accepted

Can I tail the log of ONE container in a docker-compose group?

The docker-compose binary only commands the docker daemon. You can simply use the docker command to get what you need. Use docker ps or docker ps -a to find out about your containers. Use docker logs ....
  • 193
2 votes
Accepted

Prometheus with remote_write and backups

1) Do I need to back up the prometheus TSDB if data is being pushed into remote storage? You don't have to. Or will it attempt to query the long term storage for instance if it does not have ...
  • 3,904
2 votes

GitLab and Prometheus failure

The issue is gitlab runs Prometheus in a pyvenv setup which I can't find where it is. You can always pip install leveldb And then run the command.. it should work
  • 21.9k
2 votes
Accepted

How to monitor downtime of a remote third party MySQL Server?

A simple script to bind to the database works for me when I need to monitor remote MySQL instances. I prefer Python because it integrates well with my monitoring system, but any script that binds to ...
  • 36
2 votes
Accepted

What alerting setup can give me an overview of my Prometheus alert states?

Without getting prometheus-specific, a good monitoring system informs you both ways: it tells you what it believes you need to know, and it lets you find what you want to know as quickly and ...
  • 78.5k
2 votes

Prometheus 1.5.2 behind Apache 2.4 reverse proxy

I'm running newer versions of prometheus now, I had this in the 1.X series as well, -web.external-url=https://<proxyhost>/prometheus Try and see if it works in your version (I can't find ...
  • 530
2 votes

kubernetes: measuring resource/limit efficiency

So there are many possible approaches to this case. Grafana would be one of them. First of all you could use resource requests and/or limits. Basically requests are reserved amount of memory or CPU ...
  • 174
2 votes
Accepted

How to check to exist a unit systemd on the servers by Prometheus?

My solution for this is just monitoring if someone listening to some port on the server, that used by service_1.
  • 63
2 votes

How to monitor ping packet loss via prometheus exporters?

The smokeping_prober grafana dashboard has: A histogram of response times; A graph of packet loss; and A graph of latencies. It achieves the packet loss graph using the following expression: ( ...
  • 411
2 votes

Disk usage metrics missing from Prometheus node exporter

Pay your attention to /etc/default/prometheus-node-exporter: for e. g., recently I realised Ubuntu's (18.04) package cuts out by default all disks statistics: ARGS="--collector.diskstats.ignored-...
  • 9,181
2 votes

Smoothing Grafana graph

I should preface my answer by saying that I'm a Prometheus newbie getting used to PromQL myself. I should remind myself that 'rate' is something that should only be applied to counters, and not ...
  • 3,919
2 votes
Accepted

Prometheus: scrape interval is 1m, but resolution is still 15s

The resolution there is the resolution of the graph, which is automatically determined by the width of the graph and the time period it covers. It bears no relation to the scrape interval. If you ...
  • 3,904
2 votes

AWS CloudWatch metrics - are there convergence delays?

I have not found any official AWS documentation saying so, but this is true and happens for ELB/ALB at least, which I have observed. The "convergence of metrics" means it may take up to 10 ...
2 votes
Accepted

Do prometheus metrics have some sort of "freshness" lifetime?

To the title question - yes, it is documented at https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness: If no sample is found (by default) 5 minutes before a sampling timestamp, ...
2 votes
Accepted

Prometheus node_exporter metrics - huge spikes in Grafana

3173715943503 @1585311277.492 3173715937480 @1585311277.493 The values are going backwards, which is treated as a counter reset. This would usually indicate a kernel bug, however given that the ...
  • 3,904

Only top scored, non community-wiki answers of a minimum length are eligible