2

I'm using prometheus and alertmanager to post messages to slack when alerts fire. Both are self-hosted in Kube. PII concerns prevent me from using a managed instrumentation solution.

I'm looking for options that let me see the current state of all alerts at a glance. At the moment a system failure can trigger anywhere from 1-20 alerts, manifesting as 1-40 slack messages (one each for alert high, alert low) and it's noisy enough that I can miss stuff.

My preference would be a tool that posts, say, one slack message every ten minutes and keeps editing it as status changes.

I've experimented with Grafana dashboards but I've found that it's not practical because it's rare for anyone to proactively check it.

How do you guys efficiently observe the health of your systems' states?

Improve this question
6
  • You appear to be confused about exactly what you want. Do you want proactive, or do you want observable?
    – womble ♦
    Sep 11, 2018 at 6:53
  • @womble As far as I know, I want a summary of my systems' alerting states in slack (as opposed to a dashboard link or 40 sequential slack messages.) But these are the only tools I've used so I may well be missing a better approach.
    – Plato
    Sep 11, 2018 at 7:22
  • You're describing an XY problem, which we're rather ill-equipped to answer. Please describe the underlying problem you're having.
    – womble ♦
    Sep 11, 2018 at 8:06
  • Alertmanager tells me one thing at a time. I want it to tell me all things at one time.
    – Plato
    Sep 11, 2018 at 20:52
  • ^ actually that's probably the X, which is why I asked for what you guys do. The Y is "I want the output of my alerting pipeline to have high SNR"
    – Plato
    Sep 11, 2018 at 23:09

1 Answer 1

Reset to default
2

Without getting prometheus-specific, a good monitoring system informs you both ways: it tells you what it believes you need to know, and it lets you find what you want to know as quickly and efficiently as possible.

My personal preference is NAGIOS, because the alerting end of it is quite sophisticated (multiple alert channels, blackout time periods, acknowledgments and escalations, etc.) and also because it has some very nice synoptic views that let me take in network health at a glance.

My personal favourites for the latter are (a) the network map in "circular balloon" mode, which lets me see at a glance the health of around a hundred nodes running nearly a thousand service checks, and (b) the "unhandled service problems" view, which gives me a detailed list of everything that's currently not right on my network.

But any tool that does both things effectively (informs you and gives you synopses) should help. Try a few, until you find one that works for you.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.