Ideally, your autoprovisioning system should hook into your deploy environment to pull the initial code for new nodes. Autoscaling Groups are great for getting instances spun up, but the code-update problem is left as an exercise for the user. Baking your code in the AMI works well, unless you're updating code multiple times a day; in that case it's best to build a separate code-deploy mechanism.
It really does depend on your code. You have a couple of ways to do it.
Pull Deploys
This is the method you described: each node monitors something and once the change-event happens it knows to pull new code. There are a bunch of ways to accomplish this.
- Cron jobs that check a file or URL every n minutes.
- Ansible or something similar publishes an updated thing and agents poll and execute the UpdateCode action.
- The code itself has hooks to poll a queue service and runs update jobs when they arrive.
The advantage here is that you can scale out your code-deploy infrastructure pretty well. Pull methods mean that the interval when your code-tier is running two versions is longer, which reduces how beefy your deploy-infrastructure has to be.
However, you said you can't tolerate multiple code versions. That's going to be an issue. Pull is probably not the method you want right now (though will eventually).
Push Deploys
This method has code pushed at the nodes. Like pull, there are a bunch of ways to do this:
- Something like capistrano where the Scaling Group is polled for an instance list and code is pushed in parallel to all nodes at the same time.
- A different mode of Ansible or something similar where an agent on each node is notified by the master node to update code right now.
- A schema-key is updated in your database, and your code is smart enough to notice this and kick off a self-update process before it does any more work.
The advantage to push is that its easier to keep a tight synchronicity in your code-base. If you really need strict one-version-only tier, its easier to do. Depending on how you've configured your scaling intervals, you can turn off part of your instances, update code on the subset, and turn them on. A schema-key in your database tells the code to not process any work and forward requests to those that do have the right key. Update the key in the database and within a few miliseconds the new-code nodes start doing all the work and the old-code nodes start forwarding. Then update code on the other tier.
If you scaling-groups can't tolerate code outage... you'll probably have to go to an A/B system for scalability groups. Update the code in Group B, flip the bit in the database, change the ELB to point to Group B. Reverse for the next update.
Which methods you use depends entirely on how tolerant your overall application is to multiple code versions running at the same time. A distributed system like this is a complex system, and state in one part of the system will be slightly different than in another part of the system no matter how hard you try. The less tolerant you are, the more outage you have to incur every time you update code. The outage may be less than a second, but it's still outage.