This looks related to this one, but it's somewhat different.

There is this WAN link between two company sites, and we need to transfer a single very large file (Oracle dump, ~160 GB).

We've got full 100 Mbps bandwidth (tested), but looks like a single TCP connection just can't max it out due to how TCP works (ACKs, etc.). We tested the link with iperf, and results change dramatically when increasing the TCP Window Size: with base settings we get ~5 Mbps throughput, with a bigger WS we can get up to ~45 Mbps, but not any more than that. The network latency is around 10 ms.

Out of curiosity, we ran iperf using more than a single connections, and we found that, when running four of them, they would indeed achieve a speed of ~25 Mbps each, filling up all the available bandwidth; so the key looks to be in running multiple simultaneous transfers.

With FTP, things get worse: even with optimized TCP settings (high Window Size, max MTU, etc.) we can't get more than 20 Mbps on a single transfer. We tried FTPing some big files at the same time, and indeed things got a lot better than when transferring a single one; but then the culprit became disk I/O, because reading and writing four big files from the same disk bottlenecks very soon; also, we don't seem to be able to split that single large file into smaller ones and then merge it back, at least not in acceptable times (obviously we can't spend splicing/merging back the file a time comparable to that of transferring it).

The ideal solution here would be a multithreaded tool that could transfer various chunks of the file at the same time; sort of like peer-to-peer programs like eMule or BitTorrent already do, but from a single source to a single destination. Ideally, the tool would allow us to choose how many parallel connections to use, and of course optimize disk I/O to not jump (too) madly between various sections of the file.

Does anyone know of such a tool?

Or, can anyone suggest a better solution and/or something we already didn't try?

P.S. We already thought of backing that up to tape/disk and physically sending it to destination; that would be our extreme measure if WAN just doesn't cut it, but, as A.S. Tanenbaum said, "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway."

locked by Tom O'Connor Aug 21 '13 at 9:15

This post has been locked due to the high amount of off-topic comments generated. For extended discussions, please use chat.

  • 1
    Out of curiosity, is the time it takes really that critical? Also, would saturating the link for the duration of a 160Gb transfer not have an impact on the rest of your network? – Bryan Feb 11 '10 at 7:48
  • 6
    I remember delivering some DLT autoloaders and a couple hundred cartridges to a Customer back in '99. We calculated the raw capacity of my car w/ around 200 DLT IV cartridges loaded in it (35GB raw capacity each) at about 6.3TB. I drove from our office to the Customer's site in about 55 mintues, giving the "Evan in a Geo Metro driving like mad down the Interstate" backup transport mechanism an effective throughput of around 118GB / min. Good throughput, but the latency was a killer... >smile< – Evan Anderson Feb 11 '10 at 8:08
  • Bryan: yes, time is critical (it takes about TWENTY HOURS with standard FTP and standard network settings), and no, there will be no problem in saturating the link, because the transfer will be scheduled in off-work time. – Massimo Feb 11 '10 at 9:03
  • Evan: that's exactly what I meant ;-) – Massimo Feb 11 '10 at 9:03
  • I've been dealing with a similar situation, with ~200GB of SQL .bak, except the only way I've been able to get the WAN link to saturate is with FTP. I ended up using 7-zip with zero compression to break it into 512MB chunks. "Compression" and "decompression" times were agreeably short; all-in-all much better than shoveling physical media across country. (The sites are on opposite coasts of the U.S.) – Adrien Jun 24 '10 at 22:30
up vote 15 down vote accepted

Searching for "high latency file transfer" brings up a lot of interesting hits. Clearly, this is a problem that both the CompSci community and the commercial community has put thougth into.

A few commercial offerings that appear to fit the bill:

  • FileCatalyst has products that can stream data over high-latency networks either using UDP or multiple TCP streams. They've got a lot of other features, too (on-the-fly compression, delta transfers, etc).

  • The fasp file transfer "technology" from Aspera appears to fit the bill for what you're looking for, as well.

In the open-source world, the uftp project looks promising. You don't particularly need its multicast capabilities, but the basic idea of blasting out a file to receivers, receiving NAKs for missed blocks at the end of the transfer, and then blasting out the NAK'd blocks (lather, rinse, repeat) sounds like it would do what you need, since there's no ACK'ing (or NAK'ing) from the receiver until after the file transfer has completed once. Assuming the network is just latent, and not lossy, this might do what you need, too.

  • uftp looks really promising, I was able to achieve 30 Mbps between two desktop computers (which are definitely not-so-great at disk performance); I'll test it on the "real" servers soon. I wasn't able to get a FileCatalyst demo license due to some bug in the registration form (it keeps saying the request numer has been already used), and fasp just doesn't offer them. – Massimo Feb 11 '10 at 10:19
  • 60 Mbps between two computers with proper disks and a big receive buffer. Great! – Massimo Feb 11 '10 at 12:00
  • I love free / open source software! >smile< I'm definitely going to give uftp a try with some stuff I'm doing. I'm wondering how it would do in a Linux-based multicast disk-imaging solution that I put together a couple of years ago using "udpcast". – Evan Anderson Feb 11 '10 at 15:05
  • a while back I asked Eventually I came to the conclusion that uftp and mrsync were the tools of choice. Please post in the comments over there if you do anything useful with uftp, as I'll be using one or the other again this year (prep for a conference). – Jed Daniels Mar 18 '11 at 6:27
  • 2
    When I was working with UFTP, UDT, and Tsunami UDP, UFTP had the worst performance of the three over all. Of course, it's probably the most mature protocol. UDT only provides a simple transfer protocol and was designed to act as a library to develop custom software and the author of Tsunami actually pointed us toward UDT since Tsunami hasn't been actively developed recently due to a lack of time. – Thomas Owens Jun 23 '11 at 14:03

Really odd suggestion this one.. Set up a simple web server to host the file on your network (I suggest nginx, incidentally), then set up a pc with firefox on the other end, and install the DownThemAll extension.

It's a download accelerator that supports chunking and re-assembly.
You can break each download into 10 chunks for re-assembly, and it does actually make things quicker!

(caveat: I've never tried it on anything as big as 160GB, but it does work well with 20GB iso files)

  • 40 Mbps between the same computers. Looks really good, too. – Massimo Feb 11 '10 at 10:47
  • 1
    replace firefox with and it's not that bad of a suggestion. – Justin Feb 15 '10 at 20:36

The UDT transport is probably the most popular transport for high latency communications. This leads onto their other software called Sector/Sphere a "High Performance Distributed File System and Parallel Data Processing Engine" which might be worthwhile to have a look at.

  • 1
    I did some work with UDT for transfers over networks with high-latency and high-packet loss. UDT is much more resilient to latency and packet loss than TCP based-protocols, especially once you get into changing the congestion control algorithm to suit your network topography. – Thomas Owens Jun 22 '11 at 13:58
  • There even is a version of rsync with UDT built in, it's called "UDR". – Max Jul 14 '16 at 20:11

My answer is a bit late, but I just found this question, while looking for fasp. During that search I also found this : , the "Tsunami UDP Protocol".

From their website :

A fast user-space file transfer protocol that uses TCP control and UDP data for transfer over very high speed long distance networks (≥ 1 Gbps and even 10 GE), designed to provide more throughput than possible with TCP over the same networks.the same networks.

As far as speed goes, the page mentions this result (using a link between Helsinki, Finland to Bonn, Germany over a 1GBit link:

Figure 1 - international transfer over the Internet, averaging 800 Mbit/second

If you want to use a download accelerator, have a look at lftp , this is the only download accelerator that can do a recursive mirror, as far as I know.

  • 1
    In the project I commented on earlier in Steve-o's answer, we benchmarked UDT, Tsunami UDP, and UFTP. We found that latency had a huge impact on performance, while packet loss did not (contrary to the Tsunami documentation). Adding 100ms of latency to the test network dropped performance of Tsunami from about 250Mbits/second to about 50Mbits/second (I believe I have my numbers and units right - it's been a while, but it was a huge drop). Adding 10% packet loss no a minimal latency network, on the other hand, only decreased performance from 250Mbits/second to about 90Mbits/second. – Thomas Owens Jun 23 '11 at 14:00

The bbcp utility from the very relevant page 'How to transfer large amounts of data via network' seems to be the simplest solution.

  • I don't think that bbcp is optimized for high latency. I'm getting ~20 MB/sec over a transatlantic link at the moment with the default settings. – Max Jul 14 '16 at 20:10

protected by Community Aug 21 '13 at 9:14

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.