26

I'm trying to transfer a set of large files internationally using SFTP, but I am finding my international partner can't get upload speeds above ~50k despite very good connections on either side. We can get multiple connections uploading at this speed (so not bandwidth?), but no single upload improves in speed, which is a problem as many files are several gb in size.

The SFTP is being hosted using the standard Apple OSX "Remote Login" SFTP system.

Is there a way to improve upload speeds, or is there a different SFTP host that would help? It's not clear to me if this is a configuration problem or an inherent limitation of the protocol.

(For security reasons I need to be using an end-to-end encrypted peer-to-peer connection -- no cloud services).

  • If you have the budget, there are commercial solutions which perform much better than TCP-based file transfer systems like SFTP. – Kenster Apr 10 '17 at 16:41
  • 4
    If it is one time-multi-gb transfer why not try an alternative to internet. – vasin1987 Apr 11 '17 at 2:43
  • 1
    A simple shell script to start N rsync transfers will easily achieve your requirements of 1. Secure transfer and 2. Maximizing your bandwidth. See here for an example of how to start N rsync transfers stackoverflow.com/a/38014502/52074 – Trevor Boyd Smith Apr 11 '17 at 3:26
  • 2
    Or just use uftp-multicast.sourceforge.net wish will encrypt and Mac out your bandwidth. – Trevor Boyd Smith Apr 11 '17 at 3:30
  • 4
    Contrary to your last sentence, cloud service should be okay if you encrypt the file locally, transfer it via cloud, then decrypt locally 8at the other end), which would still mean end-to-end encryption. (You may want to add some short feedback about successful reception). You use sftp encryption to prevent attacks by someone able to sniff all your traffic. Hence just giving them the encrypted data is not worse than assuming they might get it anyway. – Hagen von Eitzen Apr 11 '17 at 6:32
29

With OpenSSH sftp client (which you seem to use), you can use:

  • -R switch to increase request queue length (default is 64)
  • -B switch to increase read/write request size (default is 32 KB)

For a start, try to double both:

sftp -R 128 -B 65536 user@host

It probably does not matter much, which of them you increase.

Increasing either should help to saturate your high-latency connection. With the above settings, it will keep 8 MB worth of data flowing in the pipe at any time (128*64K=8M).

Note that this helps with big file transfers only. It won't have any effect, when transferring a lot of small files.


For some background and a discussion about other (GUI) SFTP clients, see the "Network delay/latency" section of my answer to Why is FileZilla SFTP file transfer max capped at 1.3MiB/sec instead of saturating available bandwidth? rsync and WinSCP are even slower.

4

You could try and enable compression, and see if that helps.

From man sftp:

-C Enables compression (via ssh's -C flag).

And from man ssh:

-C Requests compression of all data (including stdin, stdout, stderr, and data for forwarded X11, TCP and UNIX-domain connections). The compression algorithm is the same used by gzip(1), and the “level” can be controlled by the CompressionLevel option for protocol version 1. Compression is desirable on modem lines and other slow connections, but will only slow down things on fast networks. The default value can be set on a host-by-host basis in the configuration files; see the Compression option.

It rather sounds as though the connection might be rate limited at some point along its path (or rather, that seems to me the simplest explanation for your 50kB/s per connection, but multiple such connections being possible), although it might not be a bad idea to make sure the disks on either side aren't a factor.

You could also run a quick pcap to see if there are any 'obvious' issues (such as a large number of retransmits) - but unless you had some confidence you would be able to address this, I would probably just see if enabling compression would help.

  • Thanks! Unfortunately the files are pre-compressed, so I doubt that'll do anything... :/ – nick_eu Apr 10 '17 at 16:24
  • Compression does not speed up things here even if the data would not be compressed. It is too large overhead of CPU time (and delay) so it does not make sense these days. – Jakuje Apr 10 '17 at 18:36
  • 1
    If the bottleneck is the network then a little more CPU on either side shouldn't slow anything down @Jakuje, unless the box is unable to compress at 50kB/s, which shouldn't be an issue. – Ben Apr 10 '17 at 21:47
  • @Ben The question clearly states that the network is not a bottleneck. – Jakuje Apr 10 '17 at 21:49
4

I'm trying to transfer a set of large files internationally using SFTP

It hasn't been mentioned as an answer yet, but when transferring multiple files over a high-latency link, there's one really simple solution to get better performance:

Transfer multiple files in parallel.

And it is a solution that you even mentioned in your question. Use it.

Basically, the TCP protocol doesn't handle connections with a large bandwidth-delay product very well - a single connection can't keep enough data moving at any one time. See https://en.wikipedia.org/wiki/TCP_tuning

Since each connection is limited by the TCP protocol, just use more connections.

3

Speed up sftp transfers

Assuming your issues are network tuning and/or throttling per TCP connection, take a look at sftp using the lftp mirror subsystem

Network tuning on each end is a much bigger topic and would require a lot of back and forth, pushing the topic outside of the scope of ServerFault. For individual connections, the compression mentioned by iwaseatenbyagrue may help either way. This assumes the remote end allows compression.

3

(You mention "high latency" in the question title, but not in the body text. Have you measured the actual latency, and what are the results?)

There's a patch to OpenSSH that explicitly improve throughput on a high-latency network link: HPN-SSH: (emphasis mine)

SCP and the underlying SSH2 protocol implementation in OpenSSH is network performance limited by statically defined internal flow control buffers. These buffers often end up acting as a bottleneck for network throughput of SCP, especially on long and high bandwith network links. Modifying the ssh code to allow the buffers to be defined at run time eliminates this bottleneck. We have created a patch that will remove the bottlenecks in OpenSSH and is fully interoperable with other servers and clients. In addition HPN clients will be able to download faster from non HPN servers, and HPN servers will be able to receive uploads faster from non HPN clients.

So, try to compile and use HPN-SSH on the receiving side, and see whether it improves your transfer speed.

  • Thanks! I have not actually measured, I'm now embarrased to admit, but I'm going halfway around the world into a country with so-so internet, so I'm guessing I'm right. :) Patch sounds very useful! – nick_eu Apr 11 '17 at 20:28
  • @nick_eu I have seen anecdotes that scientists would use HPN-SSH to transfer large amounts of scientific data across the Atlantic. Sounds like it should be perfect for your use case. – twisteroid ambassador Apr 12 '17 at 6:33
0

Not sure if this is an option for you, but have you tried pulling vs pushing the data to the international site? As well as either at different times to see if its an issue with contention for network resources?

  • great idea, will try. – nick_eu Apr 10 '17 at 21:07
0

We can get multiple connections uploading at this speed (so not bandwidth?)

It sounds like a configuration issue - either deliberately (as a way of upselling services without having to make any extra provision) or by accident (e.g. broken window scaling or overzealous traffic control). While you could parallelize the transfers you've told us nothing about what is at the other end of the connection or if its worth developing some simple scripts to handle sharding/reconstitution of files.

Tuning the queue size and compression are unlikely to have any significant impact, unless the cause is very badly written software (and openSSH does not into this category - not much point in using openssh with a longer request queue/larger block size unless the latency is over 250msec. You might consider trying with different clients from different locations to rule out a problem with the server.

My first call would be to identify which provider is to blame for the problem, ask them to fix the problem or switch to a different provider.

  • Sorry, should have been more clear. There is no "provider" -- I'm hosting on my own desktop, and a colleague is trying to connect from their computer. The colleague is just opening an ssh session (not sure of protocol but can check) and using put – nick_eu Apr 10 '17 at 21:06
  • @nick_eu he is talking about the internet providers. – Džuris Apr 11 '17 at 3:38
  • It sounds like a configuration issue No. It's not a configuration problem. The TCP protocol itself doesn't perform well on connections with a large bandwidth-delay product. Basically, if the connection is such that a lot of data can be in flight at a time, the TCP protocol itself can't keep that much data moving at any moment in time. Which is why parallel TCP connections do work to improve data transfer rates. – Andrew Henle Apr 11 '17 at 11:30
  • "doesn't perform well on connections with a large bandwidth-delay product" - please read RFC 1323 (from 1992) and 7323 (replaced 1323 in 2014) – symcbean Apr 11 '17 at 16:07
  • @symcbean Then explain the OP's We can get multiple connections uploading at this speed (so not bandwidth?), but no single upload improves in speed That's a classic symptom of TCP over a connection with extreme latency - all the TCP extensions can do is mitigate the problem somewhat as they can't address the fundamental problems with the protocol itself. And good luck with identify which provider is to blame for the problem, ask them to fix the problem while trying to "transfer a set of large files internationally". – Andrew Henle Apr 11 '17 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.