0

We have a lot of CloudWatch Alarms set on a variety of metrics.
I've noticed that when alarms trigger, the data described in the alert email don't always match anything in the CloudWatch Metrics view.

Example:

This morning we got an alert email:

You are receiving this email because your Amazon CloudWatch Alarm
 "HIGH-LBTargetGroup-production-www-1234567890-38-elevated-TargetResponseTime"
 in the US East (N. Virginia) region has entered the ALARM state, because
 "Threshold Crossed: 1 out of the last 1 datapoints
 [15.803533 (22/05/19 14:34:00)] was greater than the threshold (15.0)
 (minimum 1 datapoint for OK -> ALARM transition)." at "Wednesday 22 May,
 2019 14:35:52 UTC".

...

Alarm Details:
- Name:           HIGH-LBTargetGroup-production-www-1234567890-38-elevated-TargetResponseTime
- Description:    The average total time it takes for the server to
 respond to a http request.
- State Change:   OK -> ALARM
- Reason for State Change: Threshold Crossed: 1 out of the last 1 datapoints
 [15.803533 (22/05/19 14:34:00)] was greater than the threshold (15.0)
 (minimum 1 datapoint for OK -> ALARM transition).
- Timestamp:      Wednesday 22 May, 2019 14:35:52 UTC
- AWS Account:    555555555555

Threshold:
- The alarm is in the ALARM state when the metric is GreaterThanThreshold
 15.0 for 60 seconds. 

Monitored Metric:
- MetricNamespace:   AWS/ApplicationELB
- MetricName:        TargetResponseTime
- Dimensions:        [TargetGroup = targetgroup/production-www/111155559999dddd]
                     [LoadBalancer = app/production-www/ffffbbbb77773333]
- Period:            60 seconds
- Statistic:         Average
- Unit:              Seconds
...

But when I follow up in the Console, I don't see data-points that match what's cited in that email:

enter image description here

Here's the "source" for that image:

{
  "title": "HIGH-LBTargetGroup-production-www-1234567890-38-elevated-TargetResponseTime",
  "period": 60,
  "region": "us-east-1",
  "metrics": [
    [
      "AWS/ApplicationELB",
      "TargetResponseTime",
      "TargetGroup",
      "targetgroup/production-www/111155559999dddd",
      "LoadBalancer",
      "app/production-www/ffffbbbb77773333",
      { "stat": "Average" }
    ]
  ],
  "view": "timeSeries",
  "stacked": false
}

There are a few places in the console where one can see a similar graph, and they all tell basically the same story: The average response time at 14:34 was about 1.6 seconds, well below the 15-second threshold needed for the alarm in question.

This isn't a units issue, if the threshold were accidentally 1.5 seconds we'd be getting many of these emails every day.

If I switch to looking at the "Maximum" instead of "Average", then I see a pronounced spike exactly matching the datapoints cited in the email, but the alarm is clearly configured to watch the "average".

Hypothesis:

I'm pretty sure that each metric can contain datapoints from multiple "nodes" (the nature of a node would depend on the metric I guess). All of the console graphs are aggregating these streams in accordance with the specified aggregation, Maximum, Average, etc.
It seems like the Alarms aren't waiting for all nodes to report in for each minute before deciding to trigger (possibly because no list of "all nodes" is avaliable), so when the Alarm's window is set to 1 minute, the effective aggregation function is really "First".

Questions:

  • Is the above theory correct?
    • If it's (even partly) wrong, then what's actually going on?
    • If it's right, is there a way to view the un-aggregated data, either as a scatter-plot or as individual lines in a line-graph?
  • Clearly I could just always use a 2-minute or larger window, but is there anything else I could be doing to avoid these kinds of false positives?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.