File Transfer Alternatives

Question

Right now we're using the enterprise product from Aspera. We've got a license that covers only 10megabits of throughput per connection. We're looking at migrating away from Aspera as their licensing costs are a killer to get us up to 100megabits. I've been looking around and cant really find any alternatives. Right now we transfer something along the lines of 500gb a day in and out. I'd prefer something from the FOSS camp but I'm not against purchasing something Windows based.

To provide you some background on our hardware config.

(2) BL495c Blade Servers - 2x 6 core Opterons, 128gb RAM, Storage is all on HP MSA2300 FC arrays, about 10TB of data in total. Everything is running on Hyper-V Server 2008r2.

So with all of that out there the question really is:

What software/methods do you use to provide highspeed file uploads/downloads to your clients?

Edit: I'm dealing with mostly non technical end users who all live in the Windows world. So I'd like something browser based with a fairly simplistic user front end. The back end can be as technical as necessary. I think the only real requirements I've got are performance capability of approx 100megabit file transfers, user quotas, and possibly connection throttling.

Answer 1

Have you considered other commercial accelerated file transfer solutions?

Take a look at FileCatalyst www.filecatalyst.com it will give you the same speed advantage at a more affordable cost.

John

Answer 2

We use SFTP/SCP or rsync-over-SSH, but that depends on the client having the relevant client/server software at there end.

It goes as fast as out links will allow, with rsync interrupted transfers are restartable, and it is all nicely scriptable (assuming you know your bash/equivalent scripting and related tools) so things can be completely automated. You'll want to rent a server somewhere to host the files rather than transferring them direct from you to the client unless you have a 100Mbit pipe locally, but that won't be expensive unless you need significant SLAs (remember to use encrypted filesystems and other such precautions if the information is sensitive, or course).

This is bit of a DIY solution though. If you are looking for something that presents itself with a friendlier face to your clients (something akin to dropbox for example) then you'll need to add some details about what sort of thing you are looking for to your question so you get relevant answers. Remember that most of use won't have used Aspera so won't be familiar with it, and even those that do use it won't know which features you find particularly useful (and which are no concern to you) if you don't list them.

Edit:

This tool was recently on HN (here), which looks like it might be worth you investigating if you do have a fat outgoing connection. Obviously you'll have to look into the security side of things, such as keeping your clients separate from each other by some means (as it seems geared around synchronising personal files between locations, rather than sharing them with clients). Caveat: I've not tried it at all myself as yet.

Answer 3

Something from HPC that is used if your data sets are approaching PB size and your pipes are fat but have high RTT: GridFTP - parallel TCP streams, can be striped over multiple servers. Maybe a bit of an overkill for your application but it won't get faster than GridFTP. Depending on your security policies it may require a full blown X509 PKI.