5

I use EBS storage on a Debian instance. I set the instance to not terminate at shutdown.

I'm wondering what happens in case of Hardware failure (RAM, CPU, HD etc).

  1. which type of Alarm should I configure to be notified? Can I rely on "StatusCheckFailed"?

  2. Should I expect a restart/reboot on a different Hardware automatically done by AWS team? If not, what are the steps I have to follow to restart my instance on a different Hardware? How long does it take?

  3. Can I safely assume I will NOT loose my data (/var/www etc)? Currently if I stop and start everything is OK, but I am not sure if I can rely on it

  4. In case of Hard Disk failure, is it transparant because AWS uses RAID or whatever? or do I also have to be notified and maybe manually restart from a previous snapshot?

Being on the 'cloud', especially AWS, I was expecting it includes management of failover, with product like VMware, just restarting the VM on another HW, automatically. So I understand I have to expect fail over, but I'm looking for solutions to either automatically run the instance on another area/region when a HW failure is detected, or, if not possible, at least manually by going hrough a couple of steps?

Thanks, Rod

  • 1
    Never assume you will not loose data. – user9517 Feb 20 '16 at 13:29
  • The cloud is just a marketing term The cloud is just someone else's infrastructure that you have abdicated some responsibilities to. It is subject to all of the same reliability issues of your own hardware. – user9517 Feb 20 '16 at 13:31
  • From the 'cloud', especially AWS, I was expecting it includes management of failover, with product like VMware, just restarting the VM on another HW, automatically. So I understand, I have to expect fail over, OK, but it does not answer all my questions, how to manage it? no means to automatically run the instance on another area/region when a HW failure is detected? If not at least manually by going through a couple of steps? – Rod Feb 20 '16 at 18:21
2

AWS is unlikely to restart your instance. They give you all the tools to monitor and restart instances so they leave it to you. They may email you if you need to do something. If you stop then start your instance it will move to new hardware, but a restart will not move it to new hardware. Restarts of my Amazon Linux instance typically take a minute or so.

You shouldn't lose data from your EBS disk if EC2 hardware fails, as EBS volumes are stored redundantly within a single availability zone. EBS Snapshots are stored in S3, which stores data across three availability zones within a single region, so they're significantly more robust. Snapshots can be automated to be taken hourly, daily, weekly, etc, using a variety of tools. The first snapshot is large, subsequent are differentials are said to use relatively little space. In my experience snapshots close together use little space, but over time they do add up both size and cost, so I regularly delete snapshots I don't need.

As well as snapshots You should also take application level backups using an application like Borg Backup, Restic, or a commercial tool.

You can create an alarm in CloudWatch that reboots your instance if StatusCheckFailed is raised. The documentation with step by step instructions is here.

| improve this answer | |
4

In some cases, Amazon will notice their hardware is in a degraded state and tell you to get off of it (stop and start your instance) by a certain date or it will be stopped automatically.

In some cases, there will be no warning and it will just stop. Or not enter STOP state, and simply become unreachable. It may or may not reboot after they take care of it. Sometimes, there will be an apology mail after the fact.

I have yet to have an EBS volume fail on me (I've had many instance go weird, but not volumes), but still plan for that. I don't know what that looks like.

Setting an alarm for the Reachability status check failing is your best bet.

| improve this answer | |
  • any answer about question 2? how to recover in case of HW failure? Thanks – Rod Feb 20 '16 at 18:25
  • @Rod If Amazon has flagged a specific bit of hardware as bad, they will not allow new instances to spin up on it. In that case, a simple stop/start cycle will be sufficient to move it to healthy hardware. – sysadmin1138 Feb 21 '16 at 1:36
2

I just had an EBS fail which brought down both EC2s running a supposedly fault tolerant service on Elastic Beanstalk.

The symptoms were HTTP GET requests still worked, but POSTs failed. This means our GET-based health checks didn't detect any problems, but users could not log in as the login process used a POST.

Looking through the logs, there were many messages in /var/log/messages about I/O errors.

EXT4-fs warning (device dm-3): ext4_end_bio:314: I/O error -28 writing to inode 5905292 (offset 3198976 size 4096 starting block 1475341)
Buffer I/O error on device dm-3, logical block 1475341
EXT4-fs warning (device dm-3): ext4_end_bio:314: I/O error -28 writing to inode 5905292 (offset 0 size 0 starting block 1475340)
Buffer I/O error on device dm-3, logical block 1475340
JBD2: Detected IO errors while flushing file data on dm-3-8
EXT4-fs warning (device dm-3): ext4_end_bio:314: I/O error -28 writing to inode 5905292 (offset 0 size 0 starting block 1475341)
Buffer I/O error on device dm-3, logical block 1475341

There were messages in the nginx logs complaining that POST requests failed due to the resulting read-only state of the filesystem.

open() "/var/lib/nginx/body/0000522584" failed (30: Read-only file system)
open() "/var/lib/nginx/body/0000522585" failed (30: Read-only file system)
open() "/var/lib/nginx/body/0000522586" failed (30: Read-only file system)

What seems to have happened is the usual Linux behaviour, which is that a failed disk drops the filesystem into read-only mode, which then prevented nginx from creating any temporary files to store POST data. But GETs worked fine because reading the filesystem was still ok.

Interestingly because the health checks were reporting everything was good, Elastic Beanstalk didn't terminate and recreate any EC2 instances, even though approx 35% of requests were failing with HTTP 500 errors.

Lessons learned? Make sure your health check URLs attempt to write to the same filesystem used by other processes on your EC2, so that a failed disk will also cause the health check to fail. Otherwise the problem may not get detected automatically and could require manual intervention.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.