1
vote

We are currently use one KPI in our monitoring team - The percentage of identified incidents prior of our customers. Although this KPI provides a great visibility on the core performance of the monitoring system, I think we must extend this to cover other aspects, such as the amount of false alerts, or the monitoring system pro-activeness.

Can someone share from his experience - which KPIs do you use in order to measure your monitoring system performance? (and how you collect that data).

3
  • Can you provide some more specifics about what the application is? What the users expect? What sort of things does the business think is critical? You should probably break the question up into what to monitor vs how you report on and deal with what you get back.
    – polynomial
    Sep 3, 2011 at 5:21
  • We have an extensive monitoring environment, with over 300 different monitoring packages. Our monitoring products are consumed by the NOC team, which are using our tools in order to identify incidents in our production environment, and if needed to escalate the issues to the relevant teams. The main business goal is to provide a high uptime of the monitored applications. Sep 4, 2011 at 14:57
  • This question is off-topic under current topicality rules. Jan 22, 2015 at 3:13

2 Answers 2

1
vote

In the systems I've deployed the KPIs are usually correlations to business goals, some examples:

  • number of transactions processed
  • amount of total resources in the system (like TB of SAN, # of Machines)
  • uptime for applications
  • response time for application
  • average call wait time for a call center

The best success I've had with these came when I had meetings with executives and we agreed on how to translate their business goals into KPIs, rather than operations bringing them to the table first. Make sure the KPIs cover both business goals and effectiveness of the staff meeting those goals. To that end it is also good to track things like:

  • average length of time to resolution for internal tickets/bugs
  • average amount of time a service spends offline/amount of resources 'down'
  • response time of on-call for critical events
  • number of incidents the NOC/on-call responded to (perhaps vs what actually needed work)

For the reporting I've found it is best if you can entirely automate the display/reporting of these. I've used a script aggregating rrds/db queries (of specific services) into a single rrd that we can display to the NOC/executives in a 'dashboard'. The execs love graphs and the KPIs over time helps them see how things are performing. I've used rrdtool/drraw to build dashboards before. Recently I've started using OpenNMS to do SLA reporting, it has some great features that make them easy to create and executive consumable. In certain environments I've also found it useful to generate reports like these from two sources, for instance using Gomez or Keynote reports as well. This helps keep people honest as well as increases the amount of respect people have for the reports.

You mentioned false-alerts which is an interesting point with respect to KPIs. In most of the teams I've worked with false alerts are either the result of poorly designed/implemented monitoring or monitoring showing failures that people consider 'below the threshold' of 'actual problem'. I'd be curious what sort of false alerts you were seeing and why they were allowed to persist in the system (if they are test failures are those not seen by users and subsequently should be setup as retries in the monitoring system, etc)?

2
  • One of the main difficulties in implementing viable reports in my team, is that parts of the needed data should be collected from other teams (for example NOC), and usually in these teams, this data is collected manually (for example check box in service desk software). It is hard to ensure that this data is accurate. Sep 7, 2011 at 22:08
  • As you mentioned, the ideal practice would be that no "false alerts" will be allowed to the operational console. However in reality, the managed systems are often act differently in production than in staging environments, therefore some threshold adjustments or small monitoring rewrites are needed. I think that putting a KPI to measure this can assist us in providing a more finished monitoring product to production. Sep 7, 2011 at 22:13
1
vote

I consider it a waste to use website performance data "only" for triggering alerts. We match our performance data (measured using real browser application monitoring) with shopping cart abandonment rates and other website statistics - goal is to optimize our overall user experience. The KPI here is obviously the overall conversion rate of our website. Tricky is, of course, that website performance is only one of the factors for it.

Not the answer you're looking for? Browse other questions tagged .