Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I normally use rsync's -z option to enable compression when transferring files over the internet. However, if I am on my own (idle) gigabit LAN, do I still want compression? Or will it be faster without it? What factors does this depend on? (I haven't done any benchmarks yet).

So basically, is it faster to: compress + transfer + decompress, or just transfer uncompressed?

If your network connection is slow, it's obviously better to compress... but how about if everything is running on a gigabit ethernet network?

share|improve this question
up vote 9 down vote accepted

During an rsync transfer are you CPU bound, or is your link saturated.

  • If your link is saturated, but your CPU is idle, then compress.
  • If your CPU is maxed, and your link isn't, then do not compress.
share|improve this answer
    
i can only access CPU utilization on one end of the rsync, but that should do. i can measure upload traffic from one end, and also see overall traffic from my router. that should get me somewhere. thanks much! – Corey Goldberg Jul 18 '14 at 18:00
2  
Oh, and it is entirely possible to not be either CPU, or NETWORK bound. You could be limited by the performance of your storage subsystem. In that case, compression or not won't really make any difference. The only way to fix that is to buy faster disk. – Zoredache Jul 18 '14 at 18:02
2  
This might be overly obvious but I'll point out that you should consider what type of data you're sending and if it makes sense to compress. Sending a bunch of log files, hell yes, always compress. Sending already compressed file formats like JPEG, MP3, MKV or even ZIP? Compression during transfer is probably useless. – pzkpfw Jul 31 '15 at 20:23

I work this quite often, migrating data between servers and during impossibly-long data transfers...

The short answer is to test with your specific data... This is really easy to do. Try a LAN transfer with compression off, then try it with compression on...

In my experience with production data sets across a few environments, on a GigE or greater connection, enabling rsync compression throttled transfers to ~40 Megabytes/second (or 33% of the theoretical maximum speed of a Gigabit link).

That rate varied with the size and number of files being transferred as well (lots of small files, for instance, lowered overall rsync throughput). The storage subsystems of the sender and receivers also mattered.

Recently, I've been using a modified rsync for LAN transfers. By using the UDR library with rsync (allow UDP transfer), and disabling compression and encryption, I can get wire-speed rsync transfers with mixed file types, regardless of the number of files.

share|improve this answer

Like Zoredache said, if your link isnt saturated no -z. Also, something else you can consider to help manage long transfers is the bwlimit flag.

share|improve this answer

A few details to complement the answer from Zoredache:

  • Use top on the machines to check CPU usage. Particularly on the sending machine. Use ssh to get at the sender if it is remote.

  • Use rsync with the --progress option if you have enough big files to make it show meaningful speeds. If the files are mostly small, it may not tell you much about the network speed. In that case ...

  • Use iotop on one of the machines to see the network speed

And if in doubt, don't compress.

Most big files are usually already compressed (movies, pictures, installers, ...) so trying to compress them more is a waste of time.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.