0

I am attempting to perform a backup using rsync of some files on a Digital Ocean VPS running Ubuntu 14.04.4 LTS.

rsync is extremely slow, ranging from 20-50 kB/s, only 1% of what it should be. It is equally slow when using scp or when using rsync in daemon mode, so this does not seem to be a problem particular to rsync or ssh.

I also have access to another VPS (Linode), I get the expected transfer speeds of 2-3MB/s when transferring files using rsync from that VPS.

Also, when I transfer files using rsync directly from the Digital Ocean VPS to the Linode VPS I get normal inter-server transfer rates of 15MB/s. These factors combined indicate that ssh and rsync are working fine on all the machines.

For now I've circumvented the problem by tunneling the ssh connection via the Linode VPS, but I'm curious about what might be causing the very low transfer rates.

  • Perhaps your ISP sees it as some form of peer 2 peer downloading and throttles it. Try running it within a VPN tunnel. – JayMcTee Jun 3 '16 at 11:12
  • @JayMcTee It wouldn't seem to be an ISP issue as I have no problems when rsyncing to the other server, also for what it's worth my p2p rates are top-notch and often as high as 6MB/s – Blake Walsh Jun 3 '16 at 11:46
  • It could still be an ISP issue if their filtering/throttling varies per host/network in some way. Perhaps they try to differentiate between home nodes (from where encrypted traffic is likely to be P2P traffic), known P2P friendly hosts, and other commercial data centres, and they are classifying DO as one of the former but Linode as one of the latter. Do you get similar performance using HTTPS, copying a file over SSH without rsync (SCP, SFTP, cat-via-ssh-to-cat, ...), and so on? – David Spillett Jun 3 '16 at 13:19
  • @DavidSpillett it is the most plausible explanation so far. The idea that the IP or IP range is blacklisted (brownlisted?) by my ISP and being throttled would fit the evidence, which is impaired performance with rsync, scp, and rsync daemon. I realized that the HTTP download speed was bogus because it was going via Cloudflare servers and not a direct connection to the server. – Blake Walsh Jun 3 '16 at 15:05
  • @JayMcTee at the moment I've resorted to using an SSH tunnel via the Linode VPS which provides a decent 2-3MB/s. – Blake Walsh Jun 3 '16 at 15:37
1

When you say "mainly 15Mb tarballs", how many smaller files are there? Do you see the slow rate constantly or does it look faster on the larger files? rsync (and almost any other filesystem operation) is slower at processing many small files rather than fewer large ones due to the filesystem overhead of opening and closing each file (and any directory read/modify operations needed too). This can be exacerbated by VPS providers enforcing fair(er) IO use by throttling IO operations per second for each VM to a fixed maximum value.

To illustrate this:

  1. Create a large (say ~100Mb) file with something like:
    dd if=/dev/urandom of=/tmp/test.file bs=1024 count=102400
    and pushing just that file over rsync
  2. then repeat with many small files, for example 10240 10Kb files created with:
    for i inseq 1 10240; do dd if=/dev/urandom of=test.file.$i bs=1024 count=10; done
  3. repeat with even smaller files, 102400x 1Kb files created with:
    for i inseq 1 102400; do dd if=/dev/urandom of=test.file.$i bs=1024 count=1; done

You should see the process being far slower for the smaller files even though the cumulated contents are the same size. This will be the case even if you do everything locally rather then over the network at all (though obviously the network will magnify the effect somewhat). On a little old machine here time rsync * /location/on/another/local/drive/ gives real 0m0.683s, user 0m0.596s, sys 0m0.220s for the 1*100MB case and real 0m38.793s, user 0m4.664s, sys 0m15.657s for 102,400*1Kb - a difference of ~57x.

  • A 15MB file takes about 10 minutes at 30kB/s. I'm aware that rsync is a little slower with tiny files, but that's completely different to taking 10 minutes to transfer a 15MB file. During those 10 minutes, the progress displayed by --progress is 20-50Kb/s. It's not an average over many files, and it transfers small files at a comparable average speed. – Blake Walsh Jun 3 '16 at 11:49
  • 1
    You should include that detail in the question text, otherwise others may not see it down here in the comments section. – David Spillett Jun 3 '16 at 13:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.