0

I am putting together a service that will distribute content to other websites.

The websites will get a small javascript snippet that will retrieve content from static .json files.

I need a good hosting solution for the .json files. There are about 150000 of them and I need to update them weekly.

I need a scalable capacity for 1-5 million hits a day.

It seems like amazon s3 or rackspace cloud files would be a natural choice for hosting this, but neither of them have seem a good solution for uploading 150000 files in bulk.

What is the best way to host these files?

3 Answers 3

1

Can't you .tar.bz2 the files and have the tarball untarred after upload?

Or, if not all of the files get updated, you can also use rsync.

2
  • Neither amazon s3 or rackspace cloud files supports uploading zipped files I'm afraid, but that would solve the problem for sure.
    – Hobhouse
    Mar 8, 2011 at 11:36
  • Hmmm... does it support cron jobs? Then have a cron job pull the zipped files using wget/ftp from another server.
    – pepoluan
    Mar 8, 2011 at 11:38
1

I would definitely be looking to rsync for this. It is very fast, secure, and built exactly for this kind of use.

0

This might be a little left-field, but I believe Dropbox backs to S3, and you can share a directory to the web. Revise the JSON files by dropping them in a Dropbox folder and letting Dropbox take care of the sync. I have no idea if they find being used as a CDN is acceptable, though.

JungleDisk is another piece of software, designed for backups, that can sit back and transfer files to S3. They walk you through setting up an S3 account, and Amazon.com charges you separately.

I'm sure there is plenty other software intended to make it easy to sync files to the cloud.

If concurrency is a concern, I would maybe create a new directory for each version, bring the 150,000 files to the cloud, then point the application to the new directory after sync.

-danny

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .