I have noticed that when creating an alarm for disk usage monitoring of an ebs volume mounted using nvme on the newer ec2 instance types that the "device" is sent as a metric dimension. The problem is the the nvme device name can change when the ec2 instance is rebooted so the previously setup alarm is no longer monitoring data from the correct metric.
I'm using the CloudWatch Agent to send the metrics from the ec2 instance https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html on an ubuntu 16.04 server, the dimensions attached to the metric for disk_used_percent are:
"path", "/home", "host", "ip-xxx-xxx-xxx-xxx", "device", "nvme1n1", "fstype", "ext4"
I've found that omitting any one of these dimensions from the alarm fails to associated the alarm with the metric so I assume they are all required in order to have an alarm linked with the correct metric.
When I reboot the server the dimensions may then change to:
"path", "/home", "host", "ip-xxx-xxx-xxx-xxx", "device", "nvme3n1", "fstype", "ext4"
The device has changed and the alarm for this metric is no longer associated with any metric as the dimensions no longer match.
I've looked into the agent configuration to see if there is a way the alter the dimensions that are sent but can't find any way to do this https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html
It really only needs the
host dimensions to be sent in order to be unique so I could omit the
fstype dimensions if it is possible.
How can I have an alarm that is able to persist across ec2 instance reboots without needing to be reconfigured?