Best method for amazon ec2 load balancing

Question

If we use amazon load balancing with multiple instances, when the second instance is fired up how does it access the most recent web data/filesystem. Are you suppose to store your site files inside of buckets and somehow setup apache to access the buckets?

Thanks for any input.

Answer 1

I think I'd probably do this with GlusterFS. Set up a cluster of storage servers, then mount the assets within the storage server on the scaling frontends, and serve from there.

Answer 2

I have used CloudInit scripts in the past to sync content from a central source to each instance as they boot, before the webserver starts.

Another option might be to use an EBS-based instance that has the content pre-loaded in the EBS snapshot.

Answer 3

I'm using this method:

• Every X minutes another instance makes a snapshot of the files to serve
• When the new instance is fired up, the payload script download the snapshot and copy the files to the web directory.
• Then, the new instance rsync with another instance to update the new or modified files.
• Finally the web server starts.

Also you can use EBS for the snapshots. I'm using compressed snapshots because there are many files in my web directory, the rsync would take a lot of time to copy all the files.

Answer 4

The normal cloud architecture doctrine would say that you should pull data off to a separate tier - S3 for blobs, SimpleDB for non-relational data, RDS for relational data, etc. - and that the scaling front ends should not have data on them.

EBSes and snapshots are also an option, it depends how frequently you're going to be changing the files. If users are contributing the data/files, you almost certainly need to go to a shared repo per the above. But if it's just you, then other stuff works fine.

If you do, you have to deal with replication complexity. Via cloudinit, or a dedicated pull provisioning mechanisms like chef or puppet, you can do a pull sync. The issue here is that when you want to change the content, you have to push it to all the servers as well (or rely on scheduled pulls). Which for static Web content is probably fine; once you want to manage apps across servers too it gets touchier, and also depends on whether you're changing the files about every month or about every five minutes.

We use an orchestration mechanism combined with push sync. When a new server comes up it registers and gets an immediate push of current content; then when we push new content we push it to all active servers. This has the benefit of using the same channel for initial seeding at provisioning time and later changes. Some folks hack chef/puppet to do similar (or augment them with a dedicated push mechanism like capistrano).