Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We need to set up a website which makes two or three large files publicly available - the files will be 1 or 2 terrabytes each. Although they will be public, in practise I expect only a relatively small number of scientists will want to download them. What is the best way to allow this?

I've had a quick talk to a web-hosting provider (rackspace) and they suggested a hybrid solution.

  • An entry-level managed server (we predict fairly low traffic for the website, but we do need to install some custom CGI software).
  • Some cloud storage which hooks into Limelight Networks. This would host the large files, for download by FTP.

It sounded OK to me but I know relatively little about server administration. Does it make sense?

Thanks in advance, Mark

share|improve this question
us mail would be faster if they really need the whole 1 to 2 TB file. Even if you have it spread via torrents the recipients pipe will be the limiting factor! – tony roth May 5 '10 at 16:13
I'm kinda' wishing for more detail about the content and resources. Like, is he distributing from a university or a business? Is this data that can be chopped up? What kind of platform are we talking about (Windows/Unix) as this affects tools available...are your clients likely to be home users, or other university/businesses with larger bandwidth available? – Bart Silverstrim May 5 '10 at 18:13
since you are doing research ... do you have access to Internet2? – Zypher May 5 '10 at 19:02
I know this seems like an odd question -- but could you tell me what field the data is for & if you're associated with a university? There are "Domain Archives" and "Institutional Repositories" that might be willing to host the data for you if we can find a good match. – Joe H. May 5 '10 at 19:53
@MarkJ -- as you're in the UK, you might want to take a look at ... although they're more a registry than a repository, you might be able to identify a group in your field that'd be willing to host the data files. (For solar physics data, my group's been known to make gifts of hardware (that might be limited to US orgs.) or host it). For astronomy in the UK, ; for planetary data, ask an ESA member at . – Joe H. May 6 '10 at 15:23

One or two terabyte files?

Wow...well, yeah, if they're public files, using a content delivery network to distribute it would make sense. You might also consider, if other organizations out there are willing to use it and it's useful information, hosting it as a torrent, since that is kind of optimal for spreading large files over multiple sources and acts as a kind of built-in anti-corruption check. It would suck for someone to download a terabyte of data and have the MD5 show it's corrupted.

share|improve this answer
Torrents are a great idea and they're meant for similar circumstances, but the problem is that they generally require multiple users to be downloading the torrented files at the same time. MarkJ's description mentions that only a few people will be downloading the files. On the other hand, one possible consideration is Amazon's S3.… contains a good description of how to use the feature. Beware, however, of the GB-month storage cost. – blueadept May 5 '10 at 15:41
@blueadept-yeah, that was the caveat of having other organizations that find it useful. They would have to be willing to keep the file available with a torrent client running. Alternatively make deals with some others to help distribute it with torrents. Even a CDN just means that they'd be downloading from fewer hops, not multiple streams synchronously as torrents would do. – Bart Silverstrim May 5 '10 at 15:46
The costs are prohibitive for those systems. And p2p's ability to deal with recombining from lots of chunks is still an advantage even if there aren't multiple seeders. (and this is the type of data that you download and don't delete for months or years, so you might still have seeders.) – Joe H. May 5 '10 at 16:31
+1 This is the real answer here. A torrent gives you a) verification of data (data integrity) b) lifted bandwidth limits (by re-using peer bandwidth) c) set-and-forget. Sneakernetting a bunch of hard drives is good too, but this might be cheaper, and just as effective. – Avery Payne May 5 '10 at 18:46
Thanks, it sounds like a good idea, but I don't think we could find enough users willing to join in with hosting the torrents in our case – MarkJ May 6 '10 at 4:59

There are people with experience in serving stuff similar to what you're asking for.

If you're working at a NASA center, you'll need to get a waiver to be allowed to use peer-to-peer; this goes for both the server and the users, so only making the data available via p2p might make it effectively inaccessible to some scientists (unless they're willing to go through.

Personally, when people ask for large quantities of our data (it's images and data cubes, with most files are under 100MB), If it's under a few GB, I have some CGIs that will generate tarballs / zip archives on the fly. We were looking at writing our own download manager, but I'm thinking about going more generic and writing a BagIt interface to serve un-populated Bags, and a client for filling the Bags & verifying them.

For data the size you're talking about, we have people mail us hard drives, and we format them, and mail them back. Odds are, they're going to need disk space to store it when they download it, and it only happens a few times a year, so it's more effective for us than paying for more bandwidth. (we just got a shipment yesterday of 7 2TB drives for someone who wants the full data for two of the instruments whose data we archive here).

...and I also try to make sure I don't generate files larger than 2GB ... they just get too unwieldy, and you start hitting issues with older OSes and filesystems.


And if anyone has any recommendations on limiting bandwidth and connection to a given IP within Apache, I'd be grateful -- every few days I get someone from China opening up all of the available connections to suck data out of our systems. I've seen over 800 at a time. (the firewalls are managed by another department, and they'll block IPs, but not throttle)


You might also want to ask on the Earth and Space Science Informatics mailing list -- even if it's not your field, we're all interested in data distribution issues.

share|improve this answer
+1, Sneaker-netting hard drives to save on bandwidth costs for limited distribution. – Chris S May 5 '10 at 16:43
problem with hard drive distribution (never underestimate the bandwidth of a station wagon filled with backup tapes) is that people may hesitate to invest in a hard disk and shipping costs. Downloading a terabyte takes a lot of time, but the cost of a background transfer taking a couple days is probably less onerous than risking a couple hundred bucks worth of hardware just to get these files, and he said this would have a limited audience appeal. If anything, mail as blu-ray or chopped DVD's with automated recombine script... – Bart Silverstrim May 5 '10 at 18:15
They buy the disk, we pay for shipping. (even international shipping, although, that requires filling out lots of paperwork). As most people would have to buy more storage anyway to deal with the scale we're taking about, they just fork out $100-$200 to CDW, have the drive sent to us, we load it, and send it to them filled with pictures. And we can just connect it up and let it go to town overnight ... with optical media, someone's got to keep switching disks or you have to shell out for specialized hardware. – Joe H. May 5 '10 at 19:35
well, YOU guys do that, we still don't know what the OP is even distributing or what resources he has available :-) – Bart Silverstrim May 5 '10 at 19:58
@Bart : Back before 1TB drives were available, we used to mail firewire RAID enclosures, and ask them the recipient to mail back the enclosure once they had copied the data -- but no one's taken that option in more than a year, with 1-2TB drives being so cheap. The simple fact is -- $200 bucks is cheap to get the data, when scientists spend $3-5k per seat for analysis software (Matlab, IDL, Mathematica, etc.) – Joe H. May 6 '10 at 15:14

Terabyte files, as in a tebibyte, 1024 gibibytes, over HTTP? Don't do that.

I would suggest examining which platforms (operating systems) the expected consumers of these files use. If it's Windows, then the free 7-Zip can compress the file and split the resulting output file into multiple smaller (say 3.9 GiB sized) files. On Unix GNU TAR can do much the same for you; or you could use 7-Zip again, but most Unix users may not have it installed.

These smaller files can then be transferred and decompressed at the destination. Should a file part be corrupted during transfer, then only that single smaller file will need to be re-downloaded. And should the file download take days to complete, then the user can turn off their computer whenever a smaller file is fully downloaded, and resume downloading the remaining files later. Lastly, using a compressed archive gives you built-in error checking.

The downside is that during compression & decompression the users free space on their harddisks corresponding to ~2x the file size.

You can use plain FTP or HTTP to transfer the smaller files. FTP would be my pick, but less technically inclined users may not have an FTP client, and would then prefer HTTP. It might be a good idea to write a FAQ or list of common problems -- older file systems and FTP programs often can't handle files larger than 4 gibibyte (32 bit headers).

Edit: +1 for Joe H's suggestion to sneakernet the files. Sending harddisk drives via mail / courier is faster & cheaper than transmitting over Internet, unless everyone involved has big Internet pipes.

share|improve this answer
+1 for Don't do that. Don't transfer more than 4gb at a pop even over FTP – Jim B May 5 '10 at 17:23

I agree with the sneakernet suggestions (or mabye postmailnet?) mailing a harddisk (or two) can be much quicker and cheaper.

But what if the files change over time? maybe each month is a different set of files and your users want to stay updated?

in that case, the best solution would be to send by physical media the first time, and then just download the differences.

to accomplish that, there are a few obvious suggestions:

  • publish the differences, maybe using rdiff to generate binary patchfiles. cons: if the user doesn't update every time, then has to apply all the patches he missed to catch up. unless you publish differences against n-1, n-2, n-3, etc.
  • suggest your users to use rsync, that way it doesn't matter if the user wasn't up to date. cons: your server has to support rsync.
  • use zsync (my favorite): you publish both your huge files, and a 'signature' file for each. the client downloads the signature file, calculates what it will need and downloads only those parts from the big file (using HTTP range headers to do partial downloads). cons: szync website seems outdated, you'll have to test it throughly yourself.
share|improve this answer
You're right, many science products change over time, but there's a few types of changes -- appending (eg, timeseries data will continue to grow over time; most places serve out daily files, so you can just get what's new) vs. recalibration/reprocessing (ie, we realized something was wrong in the sensor/code/whatever, and re-ran everything). In the second case, there's no good solution; in solar physics, we typically serve 'level 0' data + calibration software, but for SDO the PIs are insisting on serving 'level 1' data which is going to suck if they have to recalibrate. – Joe H. May 5 '10 at 19:47
Thanks for that. The data aren't going to be updated very frequently. Probably annually, perhaps a bit more often. – MarkJ May 6 '10 at 5:02

A soft factor to consider is how to limit the downloads. I would recommend that you have a sign page that gives them the key they need to download it, and that key is valid for x days. You can let them download it again after a second sign up etc, but this will hopefully prevent people from using it as a test download file or something like that.

If there are two many keys out at once, you can have a queue, this will control the amount of simultaneous downloads.

I remember NASA's website used something like this for there large blue marble images a while back (maybe still do).

Also, if you don't use the torret solution, I would break the file into 1GB chucks. I think that is what Akami does for Microsoft's large downloads. They do it automatically, but since these are scientists you can probably have instructions on how to join them.

share|improve this answer

You'll want a CDN that offers both user-access controls and a java-based upload/download manager.

This will fix three things;

  • They'll host your content globally and from multiple resilient points - this will server your customers better.
  • Customers will have to setup accounts before downloading, this gives you traceability and ensures people don't waste bandwidth by starting downloads they have no intention of completing.
  • A multi-OS-supporting java client will use the normally less that reliable HTTP protocol in batches to fill out the full download and deal with truncated sub-transfers - normally I hate this kind of thing (think adode downloader) but they have their place for transfers this large.

So speak to the big CDNs (Akamai etc.) and ask for this ok.

share|improve this answer
Thanks, that sounds good. The hosting provider says their cloud storage hooks into Limelight so I guess it has the advantages of multiple global hosting. But the second and particularly third bullet point seem very relevant. I found this list of CDNs, is this a good list?… – MarkJ May 6 '10 at 5:08
Yeah, it's a good start. – Chopper3 May 6 '10 at 8:10

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.