In the systems I've deployed the KPIs are usually correlations to business goals, some examples:
- number of transactions processed
- amount of total resources in the system (like TB of SAN, # of Machines)
- uptime for applications
- response time for application
- average call wait time for a call center
The best success I've had with these came when I had meetings with executives and we agreed on how to translate their business goals into KPIs, rather than operations bringing them to the table first. Make sure the KPIs cover both business goals and effectiveness of the staff meeting those goals. To that end it is also good to track things like:
- average length of time to resolution for internal tickets/bugs
- average amount of time a service spends offline/amount of resources 'down'
- response time of on-call for critical events
- number of incidents the NOC/on-call responded to (perhaps vs what actually needed work)
For the reporting I've found it is best if you can entirely automate the display/reporting of these. I've used a script aggregating rrds/db queries (of specific services) into a single rrd that we can display to the NOC/executives in a 'dashboard'. The execs love graphs and the KPIs over time helps them see how things are performing. I've used rrdtool/drraw to build dashboards before. Recently I've started using OpenNMS to do SLA reporting, it has some great features that make them easy to create and executive consumable. In certain environments I've also found it useful to generate reports like these from two sources, for instance using Gomez or Keynote reports as well. This helps keep people honest as well as increases the amount of respect people have for the reports.
You mentioned false-alerts which is an interesting point with respect to KPIs. In most of the teams I've worked with false alerts are either the result of poorly designed/implemented monitoring or monitoring showing failures that people consider 'below the threshold' of 'actual problem'. I'd be curious what sort of false alerts you were seeing and why they were allowed to persist in the system (if they are test failures are those not seen by users and subsequently should be setup as retries in the monitoring system, etc)?