Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

If we use amazon load balancing with multiple instances, when the second instance is fired up how does it access the most recent web data/filesystem. Are you suppose to store your site files inside of buckets and somehow setup apache to access the buckets?

Thanks for any input.

share|improve this question
    
Have you looked at Amazon's Simple Storage Service (S3)? This is the usual answer to your question (aws.amazon.com/s3) –  voretaq7 Feb 11 '11 at 15:33
add comment

4 Answers 4

I think I'd probably do this with GlusterFS. Set up a cluster of storage servers, then mount the assets within the storage server on the scaling frontends, and serve from there.

share|improve this answer
add comment

I have used CloudInit scripts in the past to sync content from a central source to each instance as they boot, before the webserver starts.

Another option might be to use an EBS-based instance that has the content pre-loaded in the EBS snapshot.

share|improve this answer
add comment

I'm using this method:

  • Every X minutes another instance makes a snapshot of the files to serve
  • When the new instance is fired up, the payload script download the snapshot and copy the files to the web directory.
  • Then, the new instance rsync with another instance to update the new or modified files.
  • Finally the web server starts.

Also you can use EBS for the snapshots. I'm using compressed snapshots because there are many files in my web directory, the rsync would take a lot of time to copy all the files.

share|improve this answer
add comment

The normal cloud architecture doctrine would say that you should pull data off to a separate tier - S3 for blobs, SimpleDB for non-relational data, RDS for relational data, etc. - and that the scaling front ends should not have data on them.

EBSes and snapshots are also an option, it depends how frequently you're going to be changing the files. If users are contributing the data/files, you almost certainly need to go to a shared repo per the above. But if it's just you, then other stuff works fine.

If you do, you have to deal with replication complexity. Via cloudinit, or a dedicated pull provisioning mechanisms like chef or puppet, you can do a pull sync. The issue here is that when you want to change the content, you have to push it to all the servers as well (or rely on scheduled pulls). Which for static Web content is probably fine; once you want to manage apps across servers too it gets touchier, and also depends on whether you're changing the files about every month or about every five minutes.

We use an orchestration mechanism combined with push sync. When a new server comes up it registers and gets an immediate push of current content; then when we push new content we push it to all active servers. This has the benefit of using the same channel for initial seeding at provisioning time and later changes. Some folks hack chef/puppet to do similar (or augment them with a dedicated push mechanism like capistrano).

share|improve this answer
    
But what does the new instance register with, a particular server? If so, that's a single point of failure that defeats the whole of load balancing –  David Feb 14 '11 at 19:59
1  
@David an Apache Zookeeper cluster, very reliable. You could use any other message queue type thing including Amazon SQS. –  Ernest Mueller Feb 16 '11 at 19:50
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.