1

We are powering our AWS EC2 instances using cloud formation. We have 3 different stacks - testing, staging and production. Our workflow to update the images for any of the stacks is as follows:

  1. Update a 'golden master' instance
  2. Snapshot the golden master to a disk image
  3. Change the ami reference in our cloud-formation config (via a json file) for a given stack and update the stack.

This brings down the instances in the stack + re-provisions them with the new disk image.

We've had no problems with our testing or staging stacks, which contain a single ec2 instance each. Each time we update, the image is replaced no problem.

Our production stack doesn't seem to be working in the same way :-(. It contains (at least) 2 instances sitting behind a load balancer. When we update this stack in the same way, the ec2 instances are not refreshed straight away (i.e after the update is completed, the boxes are still running from the previous disk image). The good news is the new images are used when the load balancer autoscales.

Could there be a conflict between the load balancing rules and cloud formation?

Any insight would be greatly appreciated

Examples:

Testing stack : https://gist.github.com/robsquires/629fb6da2d10869363e5

Production stack : https://gist.github.com/robsquires/79de54eeb04d620d5222

  • Can you include snippets of the CloudFormation template? or a link to it? e.g. show explicitly what is different between testing, staging and production. – dialt0ne Aug 2 '13 at 13:37
  • I've added 2 examples above – Rob Squires Aug 2 '13 at 13:55
  • It can help if your testing/production stacks are identical (ie both use load balancers and multiple machines). That way you can test/debug what is actually going to happen. – Drew Khoury Aug 7 '13 at 4:00
  • Have you considered marking the current answer as correct, or commenting with more questions if you have them – Drew Khoury Nov 23 '13 at 2:13
  • AWS has released the CloudFormation UpdatePolicy with AutoScalingRollingUpdate which should makes it possible to update EC2 instances behind an ELB with zero-downtime. I assume that this would also update the AMI's correctly then. – Freenerd Feb 3 '15 at 13:09
5

This is by design and documented in the AWS::AutoScaling::LaunchConfiguration user guide:

Important

When you update a LaunchConfiguration resource, AWS CloudFormation will delete that resource and create a new one with the updated properties and a new name. This update action does not deploy any change across the running EC2 instances in the auto scaling group. In other words, an update simply replaces the LaunchConfiguration so that when the auto scaling group launches new instances, they will get the updated configuration, but existing instances will continue to run with the configuration that they were originally launched with. This works the same way as if you made similar changes manually to an auto scaling group.

I would highly recommend that in your testing stack, you create the template exactly like you have for production, only tweak the values to reduce cost e.g. MinSize, MaxSize and DesiredCapacity are 1, etc.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.