1

I'm just starting with EC2 load balancer and trying to implement autos-calling and fault-tolerance. The database is hosted in RDS, sessions are shared, files are shared using S3.

I can't seem to find the best solution for code deployments. I don't want use fabric to re-run deploy commands on all instances.

Updating code on 1 instance, creating an AMI from that and then restart instances with that AMI seems major overkill for me as we deploy multiple times a day and the whole process is a bit cumbersome.

Ideally I would like to store application code on a shared (NFS?) volume in all instances and update code on that volume during deployments. All instances can watch a particular file for changes and restart the application workers when the file is touched.

Is there a way to use NFS and auto mount shared EBS on all instances?

or is there a better way to do this?

Summary of what is desired:

I update 1 EC2 instance / NFS volume and rest pick it up without the whole process of creaing new AMI, destroying instances and creating new ones. I don't want some instances to go in limbo when application code and database schemas are not in sync. I know best practice is to write code that supports 2 consecutive DB schema changes but we really can't afford that at this point.

2 Answers 2

3

I would suggest avoiding NFS shares as they create a single point of failure for your system and NFS clients may sometimes have trouble recovering from NFS server downtime. NFS is not really the "cloud way" of doing things. The AWS way of doing data sharing between virtual machines involves the use of various decentralized AWS services.

For this particular purpose I would suggest using the AWS S3 service. Create a S3 bucket for your software releases, upload your releases there and give the virtual machines access so they can download the releases by polling the bucket. I would also suggest using IAM roles to give the production virtual servers read-only access to the S3 bucket without any hard coded API credentials. This scheme protects your releases and bucket in case an attacker manages to gain access to your servers. As there are no hard coded API credentials they cannot be reverse engineered nor abused elsewhere.

Build and development environments can be set up in a similar way -- build server or CI server can have a role for read-write access to the bucket if it resides on AWS infrastructure. If your build/CI server is elsewhere you can set up an IAM group with appropriate access rights to S3 and create an IAM user with API credentials to be used at an external site.

You still need to prepare an AMI so that it will download and configure the release as you wish and start the application, but at least you don't have to repeat this process every time you make a new release. Using some configuration management tool such as Chef, Puppet or Ansible can probably do everything you desire, but you need to allocate some time to familiarize yourself with the tools and model your environment with them.

The infrastructure (load balancers, asgs, securitygroups, roles etc) can be modeled, created and maintained with AWS CloudFormation. If your environment grows any more complex than a single ELB and a single ASG I would recommend taking a look at that. By modeling your infrastructure with CloudFormation you can quite easily create and maintain exact copies of your entire environment - one for testing/qa and one for production for example. The infrastructure description is a json document that can be kept in your VCS repository.

I hope this helps.

1

Ideally, your autoprovisioning system should hook into your deploy environment to pull the initial code for new nodes. Autoscaling Groups are great for getting instances spun up, but the code-update problem is left as an exercise for the user. Baking your code in the AMI works well, unless you're updating code multiple times a day; in that case it's best to build a separate code-deploy mechanism.

It really does depend on your code. You have a couple of ways to do it.

Pull Deploys

This is the method you described: each node monitors something and once the change-event happens it knows to pull new code. There are a bunch of ways to accomplish this.

  • Cron jobs that check a file or URL every n minutes.
  • Ansible or something similar publishes an updated thing and agents poll and execute the UpdateCode action.
  • The code itself has hooks to poll a queue service and runs update jobs when they arrive.

The advantage here is that you can scale out your code-deploy infrastructure pretty well. Pull methods mean that the interval when your code-tier is running two versions is longer, which reduces how beefy your deploy-infrastructure has to be.

However, you said you can't tolerate multiple code versions. That's going to be an issue. Pull is probably not the method you want right now (though will eventually).

Push Deploys

This method has code pushed at the nodes. Like pull, there are a bunch of ways to do this:

  • Something like capistrano where the Scaling Group is polled for an instance list and code is pushed in parallel to all nodes at the same time.
  • A different mode of Ansible or something similar where an agent on each node is notified by the master node to update code right now.
  • A schema-key is updated in your database, and your code is smart enough to notice this and kick off a self-update process before it does any more work.

The advantage to push is that its easier to keep a tight synchronicity in your code-base. If you really need strict one-version-only tier, its easier to do. Depending on how you've configured your scaling intervals, you can turn off part of your instances, update code on the subset, and turn them on. A schema-key in your database tells the code to not process any work and forward requests to those that do have the right key. Update the key in the database and within a few miliseconds the new-code nodes start doing all the work and the old-code nodes start forwarding. Then update code on the other tier.

If you scaling-groups can't tolerate code outage... you'll probably have to go to an A/B system for scalability groups. Update the code in Group B, flip the bit in the database, change the ELB to point to Group B. Reverse for the next update.


Which methods you use depends entirely on how tolerant your overall application is to multiple code versions running at the same time. A distributed system like this is a complex system, and state in one part of the system will be slightly different than in another part of the system no matter how hard you try. The less tolerant you are, the more outage you have to incur every time you update code. The outage may be less than a second, but it's still outage.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .