We are powering our AWS EC2 instances using cloud formation. We have 3 different stacks - testing, staging and production. Our workflow to update the images for any of the stacks is as follows:
- Update a 'golden master' instance
- Snapshot the golden master to a disk image
- Change the ami reference in our cloud-formation config (via a json file) for a given stack and
This brings down the instances in the stack + re-provisions them with the new disk image.
We've had no problems with our testing or staging stacks, which contain a single ec2 instance each. Each time we update, the image is replaced no problem.
Our production stack doesn't seem to be working in the same way :-(. It contains (at least) 2 instances sitting behind a load balancer. When we update this stack in the same way, the ec2 instances are not refreshed straight away (i.e after the update is completed, the boxes are still running from the previous disk image). The good news is the new images are used when the load balancer autoscales.
Could there be a conflict between the load balancing rules and cloud formation?
Any insight would be greatly appreciated
Testing stack : https://gist.github.com/robsquires/629fb6da2d10869363e5
Production stack : https://gist.github.com/robsquires/79de54eeb04d620d5222