How to auto replace AWS EC2 instance based on metric?

Question

I want to maintain a constant number of healthy instances running. But sometimes the application will be bugged, its resource utilization rate will become so low that it could be determined just by the CloudWatch metrics alone. Then, of course, I want those instances replaced automatically. But I can't find out how to do this. The closest I can think of is auto-scaling, but according to the Default Termination Policy, all those options seem to have nothing to do with the metric of a particular instance.

I have created AMI that is ready to go on launch. The only thing I need is, to make it automatically terminate unhealthy instance and replace it with a new one. So how can I do this? Any help appreciated.

Answer 1

Custom Instance Health Checks (bottom of the page) is one option.

You would have a separate piece of code running on the machine (or any machine really) that monitors health and runs the api call that sets the instance to unhealthy

I have another half formed idea, but I'm not quite sure how to implement this one. I was the architect for an on-premise system where we had the load balancer called into a separate web server on the instance, in our case it was a tiny custom Java web server, about 50 lines of code. It returned HTTP status codes, 200 (OK) if it's going ok or 500 (ERROR) if it needs to be terminated. I suspect something like that could be integrated with auto scaling, but I haven't done this in a while and I'm not sure how you'd integrate this with auto scaling.

Here's the command from the first idea above

aws autoscaling set-instance-health --instance-id i-123abc45d --health-status Unhealthy

Answer 2

For anyone who comes across this question:

Although I believe AWS should have included such a feature in CloudWatch, sadly I cannot find any information that suggests this is available. So I have created a bash script that queries CloudWatch API to determine the resource consumption metrics, and then set instance health accordingly, as suggested by Tim:

Preparation

1. If you haven't done so, install the AWS Command Line Interface. Also available via yum or apt.
2. Configure AWS CLI by running aws configure, fill in your API key and other settings. Important: if you intend to run the script below as root like I do, you will have to run this configuration command as root. Otherwise, the script will fail.

/root/my-health-check.sh

#!/bin/bash
# retrieve metrics starting from 20 minutes ago (3 data points)
# Note: Sometimes CloudWatch failed to gather data for a specific period,
# then the number of data points returned could be less than what we expect.
# Also, when the instance just started, there will be no data point.
start_time=$(date -d "-20 minutes" -u +"%Y-%m-%dT%H:%M:%SZ")
# retrieve metrics up to now
end_time=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
# get current instance ID [1]
instance_id=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
# get current region [2]
# This is only needed if you have multiple regions to manage, otherwise just
# specify a region via `aws configure`.
region=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | sed 's/\(.*\)[a-z]/\1/')
# save data retrieved for processing [3]
# Here I used an example of retrieving "NetworkIn" of "AWS/EC2" namespace,
# with metric resolution set to 300 (5 minutes).
# For a list of available metrics, run `aws cloudwatch list-metrics`
datapoints=$(aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name NetworkIn --dimensions Name=InstanceId,Value=$instance_id --statistics Average --start-time $start_time --end-time $end_time --period 300 --region $region --output text | awk '{ print $2 }')
# custom handler
# In this example, the health check will fail if all data points fall below
# my threshold. The health check will not fail if there is no data.
healthy=0
hasdata=0
THRESHOLD=300000
for i in $datapoints; do
    # In this case, the metric(NetworkIn) is not integer.
    if (( $(echo "$i $THRESHOLD" | awk '{print ($1 > $2)}') )); then
        healthy=1
    fi
    hasdata=1
done
if [ $hasdata -eq 1 ]; then
    if [ $healthy -eq 0 ]; then
        aws autoscaling set-instance-health --instance-id $instance_id --health-status Unhealthy --region $region
    fi
fi

The rest

4. Make the script run periodically

$ chmod +x /root/my-health-check.sh
# run the script at 0, 5, 10, 15 ... 55 of every hour
$ echo "*/5 * * * * root /root/my-health-check.sh 2>&1 | /usr/bin/logger -t ec2_health_check" >> /etc/crontab

5. Power off the instance and make an AMI. Once done, create a new auto-scaling group with the AMI. Now it should terminate itself and launch a new one if the metric does not satisfy the healthy condition. Voila!

References:

[1]: EC2 instance metadata

[2]: Get current region in AWS - StackOverflow

[3]: CloudWatch - get-metric-statistics