I have a question about how to monitor Kubernetes pods effectively. Currently, my setup is:
Prometheus ---remote-storage-adapter---> Graphite (for long-term metric storage)
With this setup and export metrics of each pod, when pods are redeployed (old Pods are deleted and new Pods are created), I face 2 issues:
New metrics (for new pods) are created, make my visualization on Grafana meaningless
Since new metrics are created, Graphite creates new files, and it fills up the disk quickly because we have many deployments per day. These files only have metrics for the life time of a Pod (usually days) but they pre-occupy disk space for 1 year since this is how Graphite works.
Anyone has any solution for this issue?
- Should I keep metrics in Pod-level or Service-level (aggregate metrics of all Pods of a service)
- How long should metrics of Kubernetes Pods be kept?
- Is Graphite a good TSDB for this task?
Thank you all in advance.