2

We have EC2 instances that are EBS based, and they contain state (not critical state, but they have ElasticSearch indexes, Redis, and so on).

So our desire is, if it fails, have a new instance to replace it as similar as possible (this means using the most recent snapshot backup of that EBS root volume).

If we were to do it manually, we would have to do this not-so-straightforward procedure:

  1. Create a new volume based on the latest snapshot of the EBS root volume;

  2. Launch a new instance;

  3. Detach the new instance root volume;

  4. Attach the volume created in step 1 as root volume for the new instance;

Alternatively, we think we could create an image (AMI) based on the latest snapshot, and launch an EC2 instance directly using that AMI, in which case it would be a 2 step solution. Is that assumption correct?

But, even if we do that way, as far as we know it can't be automated, since even if we use Load Balancer with Auto Scaling Group, the launch template can't be set to automatically choose the 'most recent' snapshot for the new instance root volume.

What would be the best practice for this scenario, given the requirements of automatically launching a new instance as similar as possible as the current one that is running if it fails?

Improve this question
6
  • AWS has services for both Elasticsearch and Redis, so you don't need to worry about this kind of thing.
    – jordanm
    Dec 13, 2018 at 14:16
  • @jordanm Yes you do if you take cost into consideration. The EC2 also has a lot of other services that are not offered by Amazon services so the question still stands.
    – sandre89
    Dec 13, 2018 at 15:21
  • You might have to use Lambda. Consider using this as well.
    – Tim
    Dec 13, 2018 at 17:25
  • Create AMI images directly from the EC2 instance. Don't go the long way through EBS snapshots: it's more work, and won't work at all for Windows instances. I don't know why people still use that method.
    – Matt Houser
    Dec 13, 2018 at 23:40
  • @MattHouser the problem is we're talking about recovering from a failure in the EC2, so I don't see any other way other than using the most recent EBS snapshot.
    – sandre89
    Dec 14, 2018 at 12:36

1 Answer 1

Reset to default
3

I've got a CloudFormation template and a couple of Lambdas doing exactly what you need. When you create the CloudFormation (CFN) stack it does:

  1. Create AutoScaling Group stack from a base AMI (e.g. Amazon Linux 2). The ASG has min=1 / max=1, i.e. a single instance ASG.
  2. Every night a Lambda snapshots that instance into a new AMI.
  3. The same Lambda updated the CloudFormation stack AMI ID parameter to the latest snapshot.
  4. CFN in turn updates the ASG Launch Configuration with the latest AMI ID.
  5. If the next day the EC2 instance dies it is restarted from the last night's snapshot, and all the last days changes are preserved.
  6. Also at night another Lambda runs that cleans up the old AMIs (e.g. after 14 days).

Check out the standard-autohealing.yml CloudFormation template from my GitHub repository.

Hpe that helps :)

Improve this answer
2
  • this indeed seems exactly what we wanted! I still have to study CloudFormation but it strikes me that you had to go through all this effort when this setup seems something everyone should want to do, right? How is everybody else handling this scenario we described? They don't have automation for recovery?
    – sandre89
    Dec 14, 2018 at 12:38
  • 1
    @sandre89 there are many ways how to auto-recover. The best is having stateless nodes where you don’t store anything valuable on them and if they crash you simply rebuild them. Or you can store your Redis cache on an EFS volume mounted to the instance and if it dies you relaunch it, mount that EFS with the latest data and off you go. Speaking of Redis - consider using Elastic Cache service that provides redundant, managed Redis service. That may save you some management overhead. So no, not everybody needs my autohealing template :)
    – MLu
    Dec 14, 2018 at 18:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.