A few times per year, I need to transfer massive files (20+GBs) over the Internet (latency 20-100ms), but with reasonably fast connections on the order of 1gbps between servers. Unfortunately, when I use rsync to do so it has all sorts of issues and repeatedly fails to saturate the connection, only using all of it for a fraction of a second at a time and often using none (< 1mbps, possibly zero) for minutes on end. Neither my hard drive nor CPU on either end are pinned at 100% usage, so neither of them can be the bottleneck. SCP behaves in the same way.

Speed tests, however, manage to fully saturate my connection for the duration of the test (much longer than rsync can). I can get some success by using HTTP over Cloudflare with a multithreaded downloader (aria2c), but it still won't saturate my connection — it simply prevents it from going near zero. Interestingly, aria2c also won't saturate my connection when not going through Cloudflare.

In my research before asking this question, it seems like TCP window may have something to do with this, but I don't have a proper understand of what that is nor do I know how to change it (the only information that I can find on the Internet is in the context of speed tests, where the speed test has a command line flag, but neither rsync nor SCP have any such flags documented in their man pages). This isn't useful, however, as I have no idea how to use this knowledge to fix my download speed.

As such, I ask, how can I easily (e.g. without mucking around with custom compilation, or paying for proprietary products), like I can with rsync, scp, or http (via nginx) now, saturate my fast connections for bulk file transfers? An ideal answer will give me a set of rsync flags that I can use, in order to fully solve this problem. A perhaps more realistic answer that would still be useful would be a particular standard Linux utility that can accomplish what I want without significant tweaking. I am sure that such a thing exists, as I am sure that I am not the only one who needs to transfer large files over the Internet.

  • Hvae you looked at HPN-SSH psc.edu/hpn-ssh? That page also links to tcp tuning guidance for LFNs. However, given that you have multiple files, simply transferring more than one at a time may be enough improvement. – Mark Wagner Dec 18 '19 at 19:32
  • even if your server has a gbit card that doesnt meaned to be that the outsite connection can handle this. speedtest arent precise as long ad you dont try to download a 10gb file or similar. have you tried netcat with tar? (google for it) or rsync with compression? – djdomi Dec 18 '19 at 21:47

Using lftp I was able to push my connection to its full capacity (900Mbps), you need something like this - lftp -e 'mirror --parallel=10 --use-pget-n=10 /var/vtemp /var/vtemp' sftp://root@XX.XX.XX.XX. The only problem I faced was that throughput degraded after running for a few hours.

  • This works wonders, thanks! – john01dav Mar 16 '20 at 8:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.