Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I normally use rsync's -z option to enable compression when transferring files over the internet. However, if I am on my own (idle) gigabit LAN, do I still want compression? Or will it be faster without it? What factors does this depend on? (I haven't done any benchmarks yet).

So basically, is it faster to: compress + transfer + decompress, or just transfer uncompressed?

If your network connection is slow, it's obviously better to compress... but how about if everything is running on a gigabit ethernet network?

share|improve this question

3 Answers 3

up vote 6 down vote accepted

During an rsync transfer are you CPU bound, or is your link saturated.

  • If your link is saturated, but your CPU is idle, then compress.
  • If your CPU is maxed, and your link isn't, then do not compress.
share|improve this answer
    
i can only access CPU utilization on one end of the rsync, but that should do. i can measure upload traffic from one end, and also see overall traffic from my router. that should get me somewhere. thanks much! –  Corey Goldberg Jul 18 '14 at 18:00
1  
Oh, and it is entirely possible to not be either CPU, or NETWORK bound. You could be limited by the performance of your storage subsystem. In that case, compression or not won't really make any difference. The only way to fix that is to buy faster disk. –  Zoredache Jul 18 '14 at 18:02

I work this quite often, migrating data between servers and during impossibly-long data transfers...

The short answer, of course, is to test with your specific data... This is really easy to do, right? Try a LAN transfer with compression off, then try it with compression on...

In my experience with production data sets across a few environments, on a GigE or greater connection, enabling rsync compression throttled transfers to ~40 Megabytes/second (or 33% of the theoretical maximum speed of a Gigabit link).

That rate varied with the size and number of files being transferred as well (lots of small files, for instance, lowered overall rsync throughput). The storage subsystems of the sender and receivers also mattered.

Recently, I've been using a modified rsync for LAN transfers. By using the UDR library with rsync (allow UDP transfer), and disabling compression and encryption, I can get wire-speed rsync transfers with mixed file types, regardless of the number of files.

share|improve this answer

Like Zoredache said, if your link isnt saturated no -z. Also, something else you can consider to help manage long transfers is the bwlimit flag.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.