Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am putting together a service that will distribute content to other websites.

The websites will get a small javascript snippet that will retrieve content from static .json files.

I need a good hosting solution for the .json files. There are about 150000 of them and I need to update them weekly.

I need a scalable capacity for 1-5 million hits a day.

It seems like amazon s3 or rackspace cloud files would be a natural choice for hosting this, but neither of them have seem a good solution for uploading 150000 files in bulk.

What is the best way to host these files?

share|improve this question

3 Answers 3

Can't you .tar.bz2 the files and have the tarball untarred after upload?

Or, if not all of the files get updated, you can also use rsync.

share|improve this answer
    
Neither amazon s3 or rackspace cloud files supports uploading zipped files I'm afraid, but that would solve the problem for sure. –  Hobhouse Mar 8 '11 at 11:36
    
Hmmm... does it support cron jobs? Then have a cron job pull the zipped files using wget/ftp from another server. –  pepoluan Mar 8 '11 at 11:38

I would definitely be looking to rsync for this. It is very fast, secure, and built exactly for this kind of use.

share|improve this answer

This might be a little left-field, but I believe Dropbox backs to S3, and you can share a directory to the web. Revise the JSON files by dropping them in a Dropbox folder and letting Dropbox take care of the sync. I have no idea if they find being used as a CDN is acceptable, though.

JungleDisk is another piece of software, designed for backups, that can sit back and transfer files to S3. They walk you through setting up an S3 account, and Amazon.com charges you separately.

I'm sure there is plenty other software intended to make it easy to sync files to the cloud.

If concurrency is a concern, I would maybe create a new directory for each version, bring the 150,000 files to the cloud, then point the application to the new directory after sync.

-danny

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.