0

I want to maintain a constant number of healthy instances running. But sometimes the application will be bugged, its resource utilization rate will become so low that it could be determined just by the CloudWatch metrics alone. Then, of course, I want those instances replaced automatically. But I can't find out how to do this. The closest I can think of is auto-scaling, but according to the Default Termination Policy, all those options seem to have nothing to do with the metric of a particular instance.

I have created AMI that is ready to go on launch. The only thing I need is, to make it automatically terminate unhealthy instance and replace it with a new one. So how can I do this? Any help appreciated.

0

Custom Instance Health Checks (bottom of the page) is one option.

You would have a separate piece of code running on the machine (or any machine really) that monitors health and runs the api call that sets the instance to unhealthy

I have another half formed idea, but I'm not quite sure how to implement this one. I was the architect for an on-premise system where we had the load balancer called into a separate web server on the instance, in our case it was a tiny custom Java web server, about 50 lines of code. It returned HTTP status codes, 200 (OK) if it's going ok or 500 (ERROR) if it needs to be terminated. I suspect something like that could be integrated with auto scaling, but I haven't done this in a while and I'm not sure how you'd integrate this with auto scaling.

Here's the command from the first idea above

aws autoscaling set-instance-health --instance-id i-123abc45d --health-status Unhealthy
4
  • Yeah, I've seen custom health check there, but the application I mentioned have no easy way to detect whether it stopped, apart from resource consumption. So I was asking because CloudWatch already gathered those metrics, I wonder if it can do this automatically. – zypA13510 Mar 11 '18 at 4:23
  • I don't know any way, but hopefully someone else will have an idea for you. – Tim Mar 11 '18 at 7:08
  • I just wrote a script that queries CloudWatch API to determine resource consumption and calls custom health check if the result is below a certain threshold. Will see if it works. – zypA13510 Mar 11 '18 at 9:57
  • 1
    Great. If it does suggest you add it here as an answer, to help others in future. – Tim Mar 11 '18 at 18:12
0

For anyone who comes across this question:

Although I believe AWS should have included such a feature in CloudWatch, sadly I cannot find any information that suggests this is available. So I have created a bash script that queries CloudWatch API to determine the resource consumption metrics, and then set instance health accordingly, as suggested by Tim:

Preparation

  1. If you haven't done so, install the AWS Command Line Interface. Also available via yum or apt.
  2. Configure AWS CLI by running aws configure, fill in your API key and other settings. Important: if you intend to run the script below as root like I do, you will have to run this configuration command as root. Otherwise, the script will fail.

/root/my-health-check.sh

#!/bin/bash
# retrieve metrics starting from 20 minutes ago (3 data points)
# Note: Sometimes CloudWatch failed to gather data for a specific period,
# then the number of data points returned could be less than what we expect.
# Also, when the instance just started, there will be no data point.
start_time=$(date -d "-20 minutes" -u +"%Y-%m-%dT%H:%M:%SZ")
# retrieve metrics up to now
end_time=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
# get current instance ID [1]
instance_id=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
# get current region [2]
# This is only needed if you have multiple regions to manage, otherwise just
# specify a region via `aws configure`.
region=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | sed 's/\(.*\)[a-z]/\1/')
# save data retrieved for processing [3]
# Here I used an example of retrieving "NetworkIn" of "AWS/EC2" namespace,
# with metric resolution set to 300 (5 minutes).
# For a list of available metrics, run `aws cloudwatch list-metrics`
datapoints=$(aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name NetworkIn --dimensions Name=InstanceId,Value=$instance_id --statistics Average --start-time $start_time --end-time $end_time --period 300 --region $region --output text | awk '{ print $2 }')
# custom handler
# In this example, the health check will fail if all data points fall below
# my threshold. The health check will not fail if there is no data.
healthy=0
hasdata=0
THRESHOLD=300000
for i in $datapoints; do
    # In this case, the metric(NetworkIn) is not integer.
    if (( $(echo "$i $THRESHOLD" | awk '{print ($1 > $2)}') )); then
        healthy=1
    fi
    hasdata=1
done
if [ $hasdata -eq 1 ]; then
    if [ $healthy -eq 0 ]; then
        aws autoscaling set-instance-health --instance-id $instance_id --health-status Unhealthy --region $region
    fi
fi

The rest

  1. Make the script run periodically
$ chmod +x /root/my-health-check.sh
# run the script at 0, 5, 10, 15 ... 55 of every hour
$ echo "*/5 * * * * root /root/my-health-check.sh 2>&1 | /usr/bin/logger -t ec2_health_check" >> /etc/crontab
  1. Power off the instance and make an AMI. Once done, create a new auto-scaling group with the AMI. Now it should terminate itself and launch a new one if the metric does not satisfy the healthy condition. Voila!

References:

[1]: EC2 instance metadata

[2]: Get current region in AWS - StackOverflow

[3]: CloudWatch - get-metric-statistics

1
  • P.S. if you only need to restart the instance (instead of terminate-relaunch), it's better to use aws ec2 reboot-instances than auto-scaling. – zypA13510 Mar 13 '18 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.