5

DANGER!

Do not run this command to 'test' it unless you are prepared for a crash and/or force-rebooting your system.

The steps I took:

  • I created a t1.micro instance on EC2 running Ubuntu 14.01 LTS.
  • I verified that both status checks passed.
  • I SSH'd into the instance.
  • I ran the fork bomb documented in Why did this command make my system lag so bad I had to reboot?.
    • My SSH session is shown below.
  • As you can see, the instance (quickly) ran out of memory, and my session terminated after a timeout.

I expected Instance Status Check to fail. However, both status checks continue to pass more than 20 minutes later. The instance is unresponsive to SSH and ping, although nmap reports that port 22 is open.

I was hoping to use the status check to determine if the instance was responsive and have its autoscaling group terminate and replace it, but it doesn't look like I'll be able to do that.

I have two questions:

  1. Why is the instance passing both status checks?
  2. Is there another solution (other than paying $18/month for a load balancer that isn't being used to balance load) to terminate instances that become unresponsive? Is there something I can do with cloudwatch alarms?
    • Ideally, I'd like to be able to have the instance report its health periodically, and if it fails to do so for a certain amount of time, terminate it (and let my autoscaling group take care of the rest).

My SSH session:

Welcome to Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-57-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Thu Jul  9 18:50:39 UTC 2015

  System load: 0.0               Memory usage: 7%   Processes:       47
  Usage of /:  16.8% of 7.75GB   Swap usage:   0%   Users logged in: 0

  Graph this data and manage this system at:
    https://landscape.canonical.com/

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud


Last login: [[redacted]]
ubuntu@ip-172-31-18-225:~$ :(){ :|: & };:
[1] 1218
ubuntu@ip-172-31-18-225:~$ -bash: fork: Cannot allocate memory
-bash: fork: Cannot allocate memory
Connection to 52.2.62.141 closed.

Edit: So, my real goal is to close the gap between what the status checks check for, and checking that my application is running. If the status checks really do check if the kernel is running properly, it seems to me that I could use a kernel software watchdog (like the softdog kernel module) to close that gap.

  • Do the status checks actually check that the kernel is running as it should?
  • If the status checks say the kernel is running, does that necessarily mean that all the kernel modules I've loaded are running properly?
4

Unresponsive != no heartbeats. The kernel is still running. AWS has no way of knowing that you've consumed all of your memory.

AWS Cloudwatch monitoring is really just the bare minimum you should do. If you need more detailed monitoring, you'll need to roll your own.

  • Right, so what would you suggest doing? I don't actually want detailed monitoring. I (effectively) want to periodically kick a watchdog from my application. Do you know of a way to do that with AWS services? – Collin Jul 9 '15 at 20:41
  • Sure, you can roll your own Cloudwatch metric and then adjust your ASG based on that. – EEAA Jul 9 '15 at 20:44
  • Would you suggest I use one metric and one alarm per instance? That seems a bit excessive, although I don't know of any better way at the moment. – Collin Jul 9 '15 at 20:58
  • 1
  • This was very helpful, but I ended up solving the problem another way (see my answer). – Collin Jul 10 '15 at 19:52
2

Since the status checks already take care of making sure the kernel is up, it's sufficient to use the softdog kernel module. Although this is a software watchdog timer, the fact that it is a kernel module means that any instance in which the watchdog itself becomes unresponsive would also be detected by the Instance Status Check performed by AWS, which in turn would terminate the EC2 instance.

Here's what I did in my setup script (this was an Ubuntu AMI):

cat >>/etc/modules <<EOF
softdog
EOF

apt-get install watchdog

cat >>/etc/watchdog.conf <<EOF
interval = 10
logtick = 60
max-load-1 = 24
max-load-5 = 18
max-load-15 = 12
min-memory = 65536
watchdog-device = /dev/watchdog0
ping = 8.8.8.8
interface = eth0
test-binary = /path/to/my/health/check/script.sh
test-timeout = 30
realtime = yes
priority = 1
EOF

...other setup stuff...

reboot

# If you don't want to reboot, you can instead do:
modprobe softdog
service watchdog restart

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.