8

I normally use rsync's -z option to enable compression when transferring files over the internet. However, if I am on my own (idle) gigabit LAN, do I still want compression? Or will it be faster without it? What factors does this depend on? (I haven't done any benchmarks yet).

So basically, is it faster to: compress + transfer + decompress, or just transfer uncompressed?

If your network connection is slow, it's obviously better to compress... but how about if everything is running on a gigabit ethernet network?

Improve this question
0

5 Answers 5

Reset to default
18

During an rsync transfer are you CPU bound, or is your link saturated.

  • If your link is saturated, but your CPU is idle, then compress.
  • If your CPU is maxed, and your link isn't, then do not compress.
Improve this answer
3
  • i can only access CPU utilization on one end of the rsync, but that should do. i can measure upload traffic from one end, and also see overall traffic from my router. that should get me somewhere. thanks much!
    – Corey Goldberg
    Jul 18, 2014 at 18:00
  • 5
    Oh, and it is entirely possible to not be either CPU, or NETWORK bound. You could be limited by the performance of your storage subsystem. In that case, compression or not won't really make any difference. The only way to fix that is to buy faster disk.
    – Zoredache
    Jul 18, 2014 at 18:02
  • 5
    This might be overly obvious but I'll point out that you should consider what type of data you're sending and if it makes sense to compress. Sending a bunch of log files, hell yes, always compress. Sending already compressed file formats like JPEG, MP3, MKV or even ZIP? Compression during transfer is probably useless.
    – pzkpfw
    Jul 31, 2015 at 20:23
11

I work this quite often, migrating data between servers and during impossibly-long data transfers...

The short answer is to test with your specific data... This is really easy to do. Try a LAN transfer with compression off, then try it with compression on...

In my experience with production data sets across a few environments, on a GigE or greater connection, enabling rsync compression throttled transfers to ~40 Megabytes/second (or 33% of the theoretical maximum speed of a Gigabit link).

That rate varied with the size and number of files being transferred as well (lots of small files, for instance, lowered overall rsync throughput). The storage subsystems of the sender and receivers also mattered.

Recently, I've been using a modified rsync for LAN transfers. By using the UDR library with rsync (allow UDP transfer), and disabling compression and encryption, I can get wire-speed rsync transfers with mixed file types, regardless of the number of files.

Improve this answer
3
  • I would assume that this could be improved by reducing the compression level to 1, or perhaps slightly higher. I've yet to find tests which discuss this option.
    – William T Froggard
    Jan 14, 2018 at 16:51
  • @WilliamTFroggard with --compress-level=1 I'm finding the gigabit connection mostly isn't saturated - getting about 50 megabytes/second, so unless your data is more compressible than mine it'll be best to run without any compression.
    – JosephH
    Aug 15, 2022 at 19:39
  • Yeah, I was wrong back then. Compression should only be used over slow links, such as VPN connections, which are often bottlenecked by something.
    – William T Froggard
    May 25 at 22:13
2

Like Zoredache said, if your link isnt saturated no -z. Also, something else you can consider to help manage long transfers is the bwlimit flag.

Improve this answer
1
  • 1
    According to the manual, bwlimit flag allows you to specify the maximum transfer rate for the data sent over the socket. Could you explain why this functionality helps manage long transfers?
    – Samuel Li
    Sep 11, 2018 at 23:33
0

A few details to complement the answer from Zoredache:

  • Use top on the machines to check CPU usage. Particularly on the sending machine. Use ssh to get at the sender if it is remote.

  • Use rsync with the --progress option if you have enough big files to make it show meaningful speeds. If the files are mostly small, it may not tell you much about the network speed. In that case ...

  • Use nload on one of the machines to see the network speed

And if in doubt, don't compress.

Most big files are usually already compressed (movies, pictures, installers, ...) so trying to compress them more is a waste of time.

Improve this answer
0

Here is linked question "Does compression option -z with rsync speed up backup": https://unix.stackexchange.com/questions/188737/does-compression-option-z-with-rsync-speed-up-backup

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.