Cloud formation - updating a stack behind an elb doesnt update the AMI

Question

We are powering our AWS EC2 instances using cloud formation. We have 3 different stacks - testing, staging and production. Our workflow to update the images for any of the stacks is as follows:

1. Update a 'golden master' instance
2. Snapshot the golden master to a disk image
3. Change the ami reference in our cloud-formation config (via a json file) for a given stack and update the stack.

This brings down the instances in the stack + re-provisions them with the new disk image.

We've had no problems with our testing or staging stacks, which contain a single ec2 instance each. Each time we update, the image is replaced no problem.

Our production stack doesn't seem to be working in the same way :-(. It contains (at least) 2 instances sitting behind a load balancer. When we update this stack in the same way, the ec2 instances are not refreshed straight away (i.e after the update is completed, the boxes are still running from the previous disk image). The good news is the new images are used when the load balancer autoscales.

Could there be a conflict between the load balancing rules and cloud formation?

Any insight would be greatly appreciated

Examples:

Testing stack : https://gist.github.com/robsquires/629fb6da2d10869363e5

Production stack : https://gist.github.com/robsquires/79de54eeb04d620d5222

Answer 1

This is by design and documented in the AWS::AutoScaling::LaunchConfiguration user guide:

Important

When you update a LaunchConfiguration resource, AWS CloudFormation will delete that resource and create a new one with the updated properties and a new name. This update action does not deploy any change across the running EC2 instances in the auto scaling group. In other words, an update simply replaces the LaunchConfiguration so that when the auto scaling group launches new instances, they will get the updated configuration, but existing instances will continue to run with the configuration that they were originally launched with. This works the same way as if you made similar changes manually to an auto scaling group.

I would highly recommend that in your testing stack, you create the template exactly like you have for production, only tweak the values to reduce cost e.g. MinSize, MaxSize and DesiredCapacity are 1, etc.