Distribute terrabyte files to the public from web server

Question

We need to set up a website which makes two or three large files publicly available - the files will be 1 or 2 terrabytes each. Although they will be public, in practise I expect only a relatively small number of scientists will want to download them. What is the best way to allow this?

I've had a quick talk to a web-hosting provider (rackspace) and they suggested a hybrid solution.

• An entry-level managed server (we predict fairly low traffic for the website, but we do need to install some custom CGI software).
• Some cloud storage which hooks into Limelight Networks. This would host the large files, for download by FTP.

It sounded OK to me but I know relatively little about server administration. Does it make sense?

Thanks in advance, Mark

Answer 1

One or two terabyte files?

Wow...well, yeah, if they're public files, using a content delivery network to distribute it would make sense. You might also consider, if other organizations out there are willing to use it and it's useful information, hosting it as a torrent, since that is kind of optimal for spreading large files over multiple sources and acts as a kind of built-in anti-corruption check. It would suck for someone to download a terabyte of data and have the MD5 show it's corrupted.

Answer 2

There are people with experience in serving stuff similar to what you're asking for.

• Visible Earth
• National Operational Model Archive and Distribution System

If you're working at a NASA center, you'll need to get a waiver to be allowed to use peer-to-peer; this goes for both the server and the users, so only making the data available via p2p might make it effectively inaccessible to some scientists (unless they're willing to go through.

Personally, when people ask for large quantities of our data (it's images and data cubes, with most files are under 100MB), If it's under a few GB, I have some CGIs that will generate tarballs / zip archives on the fly. We were looking at writing our own download manager, but I'm thinking about going more generic and writing a BagIt interface to serve un-populated Bags, and a client for filling the Bags & verifying them.

For data the size you're talking about, we have people mail us hard drives, and we format them, and mail them back. Odds are, they're going to need disk space to store it when they download it, and it only happens a few times a year, so it's more effective for us than paying for more bandwidth. (we just got a shipment yesterday of 7 2TB drives for someone who wants the full data for two of the instruments whose data we archive here).

...and I also try to make sure I don't generate files larger than 2GB ... they just get too unwieldy, and you start hitting issues with older OSes and filesystems.

...

And if anyone has any recommendations on limiting bandwidth and connection to a given IP within Apache, I'd be grateful -- every few days I get someone from China opening up all of the available connections to suck data out of our systems. I've seen over 800 at a time. (the firewalls are managed by another department, and they'll block IPs, but not throttle)

...

You might also want to ask on the Earth and Space Science Informatics mailing list -- even if it's not your field, we're all interested in data distribution issues.

Answer 3

Terabyte files, as in a tebibyte, 1024 gibibytes, over HTTP? Don't do that.

I would suggest examining which platforms (operating systems) the expected consumers of these files use. If it's Windows, then the free 7-Zip can compress the file and split the resulting output file into multiple smaller (say 3.9 GiB sized) files. On Unix GNU TAR can do much the same for you; or you could use 7-Zip again, but most Unix users may not have it installed.

These smaller files can then be transferred and decompressed at the destination. Should a file part be corrupted during transfer, then only that single smaller file will need to be re-downloaded. And should the file download take days to complete, then the user can turn off their computer whenever a smaller file is fully downloaded, and resume downloading the remaining files later. Lastly, using a compressed archive gives you built-in error checking.

The downside is that during compression & decompression the users free space on their harddisks corresponding to ~2x the file size.

You can use plain FTP or HTTP to transfer the smaller files. FTP would be my pick, but less technically inclined users may not have an FTP client, and would then prefer HTTP. It might be a good idea to write a FAQ or list of common problems -- older file systems and FTP programs often can't handle files larger than 4 gibibyte (32 bit headers).

Edit: +1 for Joe H's suggestion to sneakernet the files. Sending harddisk drives via mail / courier is faster & cheaper than transmitting over Internet, unless everyone involved has big Internet pipes.

Answer 4

I agree with the sneakernet suggestions (or mabye postmailnet?) mailing a harddisk (or two) can be much quicker and cheaper.

But what if the files change over time? maybe each month is a different set of files and your users want to stay updated?

in that case, the best solution would be to send by physical media the first time, and then just download the differences.

to accomplish that, there are a few obvious suggestions:

• publish the differences, maybe using rdiff to generate binary patchfiles. cons: if the user doesn't update every time, then has to apply all the patches he missed to catch up. unless you publish differences against n-1, n-2, n-3, etc.
• suggest your users to use rsync, that way it doesn't matter if the user wasn't up to date. cons: your server has to support rsync.
• use zsync (my favorite): you publish both your huge files, and a 'signature' file for each. the client downloads the signature file, calculates what it will need and downloads only those parts from the big file (using HTTP range headers to do partial downloads). cons: szync website seems outdated, you'll have to test it throughly yourself.

Answer 5

A soft factor to consider is how to limit the downloads. I would recommend that you have a sign page that gives them the key they need to download it, and that key is valid for x days. You can let them download it again after a second sign up etc, but this will hopefully prevent people from using it as a test download file or something like that.

If there are two many keys out at once, you can have a queue, this will control the amount of simultaneous downloads.

I remember NASA's website used something like this for there large blue marble images a while back (maybe still do).

Also, if you don't use the torret solution, I would break the file into 1GB chucks. I think that is what Akami does for Microsoft's large downloads. They do it automatically, but since these are scientists you can probably have instructions on how to join them.

Answer 6

You'll want a CDN that offers both user-access controls and a java-based upload/download manager.

This will fix three things;

• They'll host your content globally and from multiple resilient points - this will server your customers better.
• Customers will have to setup accounts before downloading, this gives you traceability and ensures people don't waste bandwidth by starting downloads they have no intention of completing.
• A multi-OS-supporting java client will use the normally less that reliable HTTP protocol in batches to fill out the full download and deal with truncated sub-transfers - normally I hate this kind of thing (think adode downloader) but they have their place for transfers this large.

So speak to the big CDNs (Akamai etc.) and ask for this ok.