Breaking down one prometheus.yml file?

Question

I am using Prometheus for our monitoring and I have a lot of configs (our prometheus.yml main config file is 8000+ lines long).

I would like to divide this out into logical groupings so that it becomes much readable. I came to know that Prometheus doesn't support this and we can use configuration management systems like Ansible.

Has anyone done this with their Prometheus config file? If so, how did you do it?

Answer 1

Ansible doesn't have a built in module for modifying YAML files. But you can use something like kwoodson.yedit to make any desired changes to the configuration file.

Answer 2

Prometheus can work with only one main config file. There is file service discovery feature, but it is only for adding targets. If you want to divide your monitored targets into logical groups you can have multiple instances of Prometheus, one for each each group. This way each group will have its own Prometheus server and thus its own config file. There are a few other advantages, ie.:

• each group is monitored independently, so ie. if you do some maintance on Prometheus server that monitors one group or it goes down for some reason, others are unaffected;
• each Prometheus instance has its own TSDB, so it helps to keep the databases small which should improve the query performance;
• you can choose an optimal location for each Prometheus instance, ie. AZ or VPC;
• you can still have a central console if you set up federation or build dashboard in Grafana to display metrics from multiple Prometheus servers.