2

Right now we're using the enterprise product from Aspera. We've got a license that covers only 10megabits of throughput per connection. We're looking at migrating away from Aspera as their licensing costs are a killer to get us up to 100megabits. I've been looking around and cant really find any alternatives. Right now we transfer something along the lines of 500gb a day in and out. I'd prefer something from the FOSS camp but I'm not against purchasing something Windows based.

To provide you some background on our hardware config.

(2) BL495c Blade Servers - 2x 6 core Opterons, 128gb RAM, Storage is all on HP MSA2300 FC arrays, about 10TB of data in total. Everything is running on Hyper-V Server 2008r2.

So with all of that out there the question really is:

What software/methods do you use to provide highspeed file uploads/downloads to your clients?

Edit: I'm dealing with mostly non technical end users who all live in the Windows world. So I'd like something browser based with a fairly simplistic user front end. The back end can be as technical as necessary. I think the only real requirements I've got are performance capability of approx 100megabit file transfers, user quotas, and possibly connection throttling.

5 Answers 5

2

Have you considered other commercial accelerated file transfer solutions?

Take a look at FileCatalyst www.filecatalyst.com it will give you the same speed advantage at a more affordable cost.

You could also look at Signiant on the commercial front.

On the open source side there are also a few options. UDT, Tsunami, GridFTP are all possible options. Also take a look at UFTP as well. I think the problem with these solutions will be the fact that you are dealing with non-technical end users. Perhaps with some scripting and one of those products running in the background you could create a working solutions.

On my blog, I compiled a comparison of all the open source file transfer products and I tried to identify which one will work best in certain situations (ex: high bandwidth, high latency scenario etc)

Here is the link to the blog article: http://www.filecatalyst.com/open-source-fast-file-transfers

John

note: This user is an employee of FileCatalyst.

6
  • 5
    If you're going to visit SF primarily to pimp your product, please stop.
    – ceejayoz
    Mar 20, 2013 at 1:37
  • 1
    CeeJayoz, there are many posts recommending commercial software solutions on SF. Should all commercial products be banned from SF? In this case my answer was accepted and I think that I was directly answering Ernie. May 22, 2013 at 19:00
  • 1
    Recommending commercial products can be fine. Read the FAQ, though. serverfault.com/faq#promotion 1) It shouldn't be your primary reason to be here. 2) You need to disclose your affiliation.
    – ceejayoz
    May 22, 2013 at 19:12
  • 3
    I was quite civil. It's nice that you mark it in your profile, but it's time to re-read the linked FAQ. "However, you must disclose your affiliation in your answers." People shouldn't need to read your profile to find out that your answers are potentially biased to a commercial solution of your own making.
    – ceejayoz
    May 22, 2013 at 20:41
  • 2
    John, the issue is more that you appear to work for the company in question. Taking a brief look at the posts you have made here, most of them seem to be mentioning the software product made by the company you work for. While this is generally acceptable if the product does answer the question, the FAQ does state that you MUST disclose affiliation. Beyond that, it also warns that "If a huge percentage of your posts include a mention of your product or website, you're probably here for the wrong reasons."
    – Rex
    May 22, 2013 at 20:48
5

We use SFTP/SCP or rsync-over-SSH, but that depends on the client having the relevant client/server software at there end.

It goes as fast as out links will allow, with rsync interrupted transfers are restartable, and it is all nicely scriptable (assuming you know your bash/equivalent scripting and related tools) so things can be completely automated. You'll want to rent a server somewhere to host the files rather than transferring them direct from you to the client unless you have a 100Mbit pipe locally, but that won't be expensive unless you need significant SLAs (remember to use encrypted filesystems and other such precautions if the information is sensitive, or course).

This is bit of a DIY solution though. If you are looking for something that presents itself with a friendlier face to your clients (something akin to dropbox for example) then you'll need to add some details about what sort of thing you are looking for to your question so you get relevant answers. Remember that most of use won't have used Aspera so won't be familiar with it, and even those that do use it won't know which features you find particularly useful (and which are no concern to you) if you don't list them.

Edit:

This tool was recently on HN (here), which looks like it might be worth you investigating if you do have a fat outgoing connection. Obviously you'll have to look into the security side of things, such as keeping your clients separate from each other by some means (as it seems geared around synchronising personal files between locations, rather than sharing them with clients). Caveat: I've not tried it at all myself as yet.

1

Something from HPC that is used if your data sets are approaching PB size and your pipes are fat but have high RTT: GridFTP - parallel TCP streams, can be striped over multiple servers. Maybe a bit of an overkill for your application but it won't get faster than GridFTP. Depending on your security policies it may require a full blown X509 PKI.

1

9 years later, the OP will have found a solution, but anyway:

Filesender is an open source PHP/javascript solution which seems to work well to let people transfer files through their browser.

We use it to send and receive files from people who are not used to FTP, particularly when the files are too large for the services they are used to like Dropbox, Wetransfer, etc.

Logged-in users can send download links to anyone, and can also generate "guest vouchers" to send to people to do uploads.

The default maximum file size is 100 GB., but it can be configured to whatever suits your needs (we increased it to 250 GB).

The source is on Github: https://github.com/filesender/filesender#readme

0

like john mention, filecatalyst is an option. Please correct me if i am wrong, but from my reading, i believe filecatalyst lacks the control of bandwidth consumption. it will consume all available bandwidth. however, file catalyst provides the ability to schedule it "off-peak" as an alternative to bandwidth management.

other solutions are open source UDT or Tsunami which filecatalyst is built upon.

martin

1
  • Martin, FileCatalyst protocol comes with bandwidth control. I'm not sure how come there is so much misconception about this. The Congestion Control as it is called in FileCatalyst is fully automated and will gracefully slow down to let other TCP traffic through. The user can adjust how aggressive the bandwidth control should be. Another wrong fact is that we don't use ANY open source projects in FielCatalyst. The entire protocol has been written from the ground up internally by us. May 22, 2013 at 18:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .