51

My Laptop and my workstation are both connected to a Gigabit Switch. Both are running Linux. But when I copy files with rsync, it performs badly.

I get about 22 MB/s. Shouldn't I theoretically get about 125 MB/s? What is the limiting factor here?

EDIT: I conducted some experiments.

Write performance on the laptop

The laptop has a xfs filesystem with full disk encryption. It uses aes-cbc-essiv:sha256 cipher mode with 256 bits key length. Disk write performance is 58.8 MB/s.

iblue@nerdpol:~$ LANG=C dd if=/dev/zero of=test.img bs=1M count=1024
1073741824 Bytes (1.1 GB) copied, 18.2735 s, 58.8 MB/s

Read performance on the workstation

The files I copied are on a software RAID-5 over 5 HDDs. On top of the raid is a lvm. The volume itself is encrypted with the same cipher. The workstation has a FX-8150 cpu that has a native AES-NI instruction set which speeds up encryption. Disk read performance is 256 MB/s (cache was cold).

iblue@raven:/mnt/bytemachine/imgs$ dd if=backup-1333796266.tar.bz2 of=/dev/null bs=1M
10213172008 bytes (10 GB) copied, 39.8882 s, 256 MB/s

Network performance

I ran iperf between the two clients. Network performance is 939 Mbit/s

iblue@raven $ iperf -c 94.135.XXX
------------------------------------------------------------
Client connecting to 94.135.XXX, TCP port 5001
TCP window size: 23.2 KByte (default)
------------------------------------------------------------
[  3] local 94.135.XXX port 59385 connected with 94.135.YYY port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.09 GBytes   939 Mbits/sec
2
  • 3
    rsync:// protocol or tunneling over SSH? There's very definite performance limitations in the latter¹.
    – ephemient
    Apr 8, 2012 at 0:00
  • Just add my one cent here. I am having the slow rsync issue too. My solution is adding -S to let the destination to create sparse files. My source contains many huge but sparse files. The could not only speed up from 30MB/s to 100MB/s, but also save my storage spaces.
    – midnite
    Mar 3 at 12:07

4 Answers 4

23

Another way to mitigate high CPU usage but still keep the functionality of rsync, is by moving from rsync/SSH to rsync/NFS. You could export the paths you want to copy from via NFS and then use rsync locally from the NFS mount to your destination location.

In one test from a WD MyBook Live network disk, one or more rsyncs from the NAS on a Gigabit network towards 2 local USB disks would not copy more than 10MB/sec (CPU: 80% usr, 20% sys), after exporting over NFS and rsyncing locally from the NFS share to both disks I got a total of 45MB/sec (maxing out both USB2 disks) and little CPU usage. Disk utilization when using rsync/SSH was about 6% and using rsync/NFS was closer to 24%, while both USB2 disks where close to 100%.

So we effectively moved the bottleneck from the NAS CPU to both USB2 disks.

3
  • 4
    Be warned, though, that NFS offers no security (ie: encryption).
    – WhyNotHugo
    Jul 8, 2013 at 2:31
  • 1
    This worked great! Now getting nearly full gigabit speeds when I was only getting ~100 Mb/s before.
    – PHLAK
    Oct 30, 2014 at 17:41
  • 3
    Could you point out how to use rsync/NFS ? I'm trying to transfer 8Tb between 2 MyCloud drives and it takes forever with rsync over ssh (4MB/sec)
    – FMaz008
    Sep 8, 2017 at 7:17
28

Reasons can include: compression, encryption, the number and size of files being copied, your source and destination systems' disk I/O capabilities, TCP overhead... These are all factors that can influence the type of transfer you're conducting.

Please post the rsync command you're using and provide details on the specifications of both computers.


Edit: Encryption is often a limiting factor in rsync speeds. You can run with ssh and a lighter-weight encryption cipher like arcfour

Something like: rsync -e "ssh -c arcfour"

Or you can use a modified rsync/ssh that can disable encryption. See hpn-ssh: http://psc.edu/networking/projects/hpn-ssh

But again, your laptop has a slow drive compared to your workstation. Writes may be blocked and waiting for I/O going to your laptop. What are your real performance expectations?

4
  • 1
    Laptops often have slower (7200 rpm - 5400 rpm) disks because they use less power. This could easily be your limiting factor depending on exactly what the rsync is doing.
    – Ladadadada
    Apr 7, 2012 at 23:43
  • 1
    thanks. For rsyncning from an dm-crypt encrypted disk attached to a atom processer to an ecryptfs ARM NAS box, this changed my transfer speed from 4MiB/s to 6MiB/s. rsync --protocol=29 -auh --progress /mnt/esata/pics/ -e "ssh -c arcfour" diskstation:/volume1/pics Better than nothing.
    – Sebastian
    May 26, 2013 at 15:28
  • This answer. Going from rsync -azP to rsync -aPe "ssh -c arcfour" boosted the transfer speed from 4MB/Sec to 25MB/Sec between two MyCloud Mirror drives. The receiving unit CPU is now maxed out. (think this mean I'm transfering as fast as the unit can write data)
    – FMaz008
    Sep 8, 2017 at 7:37
  • 4
    For anyone that finds this, arcfour was removed from OpenSSH in 2017. May 10, 2021 at 21:40
12

After some more testing, I finally found the answer myself. rsync uses tunneling over ssh by default. The crypto makes it slow. So I needed to get around that crypto stuff.

Solution 1: Setting up an rsync server

To use it via the rsync protocol, you have to set up an rsyncd server. There was an /etc/init.d/rsync script on my laptop, so I guessed, rsyncd was running. I was wrong. /etc/init.d/rsync start exists silently, when rsync is not enabled in /etc/default/rsync. Then you also have to configure it in /etc/rsyncd.conf, which is a pain.

If you get all this done, you have to use rsync file.foo user@machine::directory. Please note, that there are two colons.

Solution 2: Old-school rsh-server

However, the configuration was way too complicated for me. So I just installed and rsh-server on my laptop. Invoking rsync on the workstation with -e rexec then uses rsh instead of ssh. Which then almost doubled the performance to 44.6 MB/s, which is still slow. The speed bounces between 58 MB/s and 33 MB/s, which indicates there may be some buffer or congestion control problems. But that is beyond the scope of this question.

3
  • 2
    We use rsync extensively here and usually get full interface speed unless traversing millions of 4K files. I don't think the crypto is the problem unless you're using some seriously decrepit hardware.
    – Magellan
    Apr 8, 2012 at 15:16
  • Does a Intel Core2 Duo T8100 in an ThinkPad R61 count as seriously decrepit hardware? If not, why is rsync over ssh slower than rsync over rsh then?
    – iblue
    Apr 8, 2012 at 15:51
  • 5
    Encryption is often a limiting factor in rsync speeds, along with the number of files. The standard approaches to improving this are either to run rsync with a lighter encryption cipher like rsync -e "ssh -c arcfour" or trying a modified rsync/ssh that can disable encryption. See hpn-ssh: psc.edu/networking/projects/hpn-ssh
    – ewwhite
    Apr 8, 2012 at 17:17
6

These are a very old question and answers, but one important thing is missing: if you are copying already-compressed or encrypted data, turn off compression.

If your data is neither compressed nor encrypted, you still only want to compress it once! Rsync compresses with -z, ssh compresses with -C (might be by default). I haven't tested which is better since my data is compressed.

While I'm at it, you can turn off X forwarding and TTY allocation, resulting in:

rsync -avh -e "ssh -x -T -c arcfour -o Compression=no" $src $dst

Lastly, make sure (for example using iptraf) that you are actually using the network interface you think you are using. I have to my great surprise noted that on my OSX the outgoing ssh was binding to the IP on the default outgoing interface instead of to the IP on the interface the packets were supposed to be routed out on. My direct GB cross-connect between two laptops also connected by WiFi was not being used. After investigation, it was due to using 169.254/16, which the Mac puts on all interfaces, and the destination computer replying to ARP requests even though the request came in on a different interface.

4
  • Valid options, but I find that -x -T and -o Compression=no only had little effects on transfer speed.
    – FMaz008
    Sep 8, 2017 at 7:50
  • 8
    It's also worth mentioning that OpenSSH 6.7 disables arcfour.
    – bparker
    Mar 14, 2018 at 16:58
  • That's kind of a pity @bparker! Do we know which of the remaining available ciphers is lightest on the CPU?
    – Law29
    Mar 14, 2018 at 22:40
  • 3
    I just removed -z from command line I use on my Raspberry Pi running OpenMediaVault and I got like 50-200% improvement. Thanks! I am just copying hundreds of gigabytes of video files. And cooling fans just speedup on Raspberry Pi, meaning that stream of 40MB/s (instead of 10MB/s) overloads more than slow compression algorithm.
    – Marecky
    Dec 29, 2020 at 20:47

Not the answer you're looking for? Browse other questions tagged or ask your own question.