I'm using prometheus and alertmanager to post messages to slack when alerts fire. Both are self-hosted in Kube. PII concerns prevent me from using a managed instrumentation solution.
I'm looking for options that let me see the current state of all alerts at a glance. At the moment a system failure can trigger anywhere from 1-20 alerts, manifesting as 1-40 slack messages (one each for alert high, alert low) and it's noisy enough that I can miss stuff.
My preference would be a tool that posts, say, one slack message every ten minutes and keeps editing it as status changes.
I've experimented with Grafana dashboards but I've found that it's not practical because it's rare for anyone to proactively check it.
How do you guys efficiently observe the health of your systems' states?