1

I have noticed that when creating an alarm for disk usage monitoring of an ebs volume mounted using nvme on the newer ec2 instance types that the "device" is sent as a metric dimension. The problem is the the nvme device name can change when the ec2 instance is rebooted so the previously setup alarm is no longer monitoring data from the correct metric.

I'm using the CloudWatch Agent to send the metrics from the ec2 instance https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html on an ubuntu 16.04 server, the dimensions attached to the metric for disk_used_percent are:

"path", "/home", "host", "ip-xxx-xxx-xxx-xxx", "device", "nvme1n1", "fstype", "ext4"

I've found that omitting any one of these dimensions from the alarm fails to associated the alarm with the metric so I assume they are all required in order to have an alarm linked with the correct metric.

When I reboot the server the dimensions may then change to:

"path", "/home", "host", "ip-xxx-xxx-xxx-xxx", "device", "nvme3n1", "fstype", "ext4"

The device has changed and the alarm for this metric is no longer associated with any metric as the dimensions no longer match.

I've looked into the agent configuration to see if there is a way the alter the dimensions that are sent but can't find any way to do this https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html

It really only needs the path and host dimensions to be sent in order to be unique so I could omit the device and fstype dimensions if it is possible.

How can I have an alarm that is able to persist across ec2 instance reboots without needing to be reconfigured?

  • As an update to anyone reading this... I've been in contact with AWS support, they have acknowledged and confirmed the problem, they informed me there is an open request with the team responsible for this part of the system and it may be fixed at some point but they have no timeframe for any fix. – JamieD Jun 13 '19 at 11:23
-2

JamieD,

I was able to fulfill the purpose with bash script.

            #!/usr/bin/env bash

            function push_disk_used {
              aws cloudwatch put-metric-data \
              --metric-name "$1" \
              --namespace Test \
              --unit Percent \
              --value "$2" \
              --dimensions ClusterName=TestInstance \
              --region "$REGION"
            }

            # MongoDB Root Disk Space
            HOME_DISK_USED=$(df -H /home | awk 'NR ==2 {print $5}' | sed -r 's/%//')
            echo 'MongoDB Home Disk Used Percent:' "$HOME_DISK_USED";
            push_disk_used Home_DISK_USED "$HOME_DISK_USED"

Setup cron job to push disk_used_percent metrics and also setup alarms with following properties in cloudformation.

  Namespace: Test
  MetricName: HOME_DISK_USED
  Dimensions:
    - Name: ClusterName
      Value: TestInstance
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.