I have to copy a large directory tree, about 1.8 TB. It's all local. Out of habit I'd use rsync, however I wonder if there's much point, and if I should rather use cp.

I'm worried about permissions and uid/gid, since they have to be preserved in the copy (I know rsync does this). As well as things like symlinks.

The destination is empty, so I don't have to worry about conditionally updating some files. It's all local disk, so I don't have to worry about ssh or network.

The reason I'd be tempted away from rsync, is because rsync might do more than I need. rsync checksums files. I don't need that, and am concerned that it might take longer than cp.

So what do you reckon, rsync or cp?

  • 2
    If rsync does exactly what you want it to do, if you are quite familiar with its usage for this particular application already, and if it functions quickly enough to suit your taste, then why on earth would you want to switch? – eleven81 Jul 20 '09 at 14:40
  • Because I'm concerned that rsync will take longer than cp, since rsync does lots of checksumming that cp won't do – Rory Jul 20 '09 at 15:31
  • 1
    The cpu overhead of the checksum is small compared to the disk/network i/o. Unless the disk are on the same system and the OS can do some clever drive-drive copy in the bus controller. – Martin Beckett Jul 20 '09 at 17:07
  • 3
    Checksumming is done on files that differ at the size and timestamp check. If you're paranoid (like after a power outage during copy) you can force checksumming on all files, but on a local transfer, that's usually slower than starting from scratch. – korkman Oct 8 '12 at 22:53
  • 2
    Maybe he's curious about improving his workflow, and doesn't bury his head in the sand thinking he knows everything. This comment really annoys me. – Martin Konecny Oct 25 '12 at 1:32

14 Answers 14

up vote 187 down vote accepted

I would use rsync as it means that if it is interrupted for any reason, then you can restart it easily with very little cost. And being rsync, it can even restart part way through a large file. As others mention, it can exclude files easily. The simplest way to preserve most things is to use the -a flag – ‘archive.’ So:

rsync -a source dest

Although UID/GID and symlinks are preserved by -a (see -lpgo), your question implies you might want a full copy of the filesystem information; and -a doesn't include hard-links, extended attributes, or ACLs (on Linux) or the above nor resource forks (on OS X.) Thus, for a robust copy of a filesystem, you'll need to include those flags:

rsync -aHAX source dest # Linux
rsync -aHE source dest  # OS X

The default cp will start again, though the -u flag will "copy only when the SOURCE file is newer than the destination file or when the destination file is missing". And the -a (archive) flag will be recursive, not recopy files if you have to restart and preserve permissions. So:

cp -au source dest
  • 5
    The -u flag of cp probably isn't the best solution, as it would not detect a partially copied/corrupted file. The nice thing about rsync is that you can have it md5 sum the files to detect differences. – Chad Huneycutt Jul 20 '09 at 15:10
  • 3
    Adding -w (--whole-file) option would speed up an interrupted rsync, as it will just copy the file over instead of checksumming. – hayalci Jun 30 '10 at 19:50
  • 12
    actually, rsync detects local transfers and enables whole-file copy without checksumming automagically. – korkman Oct 8 '12 at 22:49
  • 21
    and --progress which is really handy! – Matt Nov 28 '12 at 3:20
  • 11
    -P or --progress shows progress for each file individually. It's useful for copying large files, not for many (thousands) small files as it means lots more output which you can't read. It doesn't show the overal progress of all files combined. – SPRBRN Jul 10 '13 at 8:56

When copying to the local file system I always use the following rsync options:

# rsync -avhW --no-compress --progress /src/ /dst/

Here's my reasoning:

-a is for archive, which preserves ownership, permissions etc.
-v is for verbose, so I can see what's happening (optional)
-h is for human-readable, so the transfer rate and file sizes are easier to read (optional)
-W is for copying whole files only, without delta-xfer algorithm which should reduce CPU load
--no-compress as there's no lack of bandwidth between local devices
--progress so I can see the progress of large files (optional)

I've seen 17% faster transfers using the above rsync settings over the following tar command as suggested by another answer:

# (cd /src; tar cf - .) | (cd /dst; tar xpf -)
  • 1
    I am having following error: rsync: --no-compress: unknown option @Ellis Percival. – alper Mar 3 at 16:31
  • This is lightening fast. Faster to do this than rm -rf /src/. – dgo Jul 18 at 15:14
  • Like @alper, --no-compress wasn't an option for my version of rsync (in CentOS 7); I used --compress-level=0 instead. – Paul Jul 30 at 23:25

When I have to copy a large amount of data, I usually use a combination of tar and rsync. The first pass is to tar it, something like this:

# (cd /src; tar cf - .) | (cd /dst; tar xpf -)

Usually with a large amount of files, there will be some that tar can't handle for whatever reason. Or maybe the process will get interrupted, or if it is a filesystem migration, the you might want to do the initial copy before the actual migration step. At any rate, after the initial copy, I do an rsync step to sync it all up:

# cd /dst; rsync -avPHSx --delete /src/ .

Note that the trailing slash on /src/ is important.

  • 6
    +1 I've found tar to generally be faster for large copies than rsync. I like the idea of finishing off with a final rsync, too. – Geoff Fritz Jul 20 '09 at 16:14
  • 2
    tar is a good choice if the dest dir is empty. Although my way would be: cd $DSTDIR; tar c -C $SRCDIR . | tar – asdmin Jul 20 '09 at 19:39
  • 19
    That's the beauty of this method. You do not need double the space because you never actually create an intermediate tar file. The tar before the pipe packs the data and streams it to stdout, and the tar after the pipe grabs it from stdin and unpacks it. – Chad Huneycutt May 10 '12 at 0:45
  • 4
    I did a cp -a for a 12gb transfer, and this method for a 42gb transfer. The tar method took about 1/4 the time. – NGaida May 23 '14 at 17:20
  • 3
    I also put pv in the middle to be able to watch the progress, estimating the size of all the data using df. I also used --numeric-owner, as the source disk was from another system and I didn't want tar to mess the owners: tar -C /old-path --numeric-owner -S -c . | pv -tpeba -s 100G | tar -C /new-path --numeric-owner -S -xp – Petr Pudlák Nov 5 '16 at 16:47

rsync

Here is the rsync I use, I prefer cp for simple commands, not this.

$ rsync -ahSD --ignore-errors --force --delete --stats $SRC/ $DIR/

cpio

Here is a way that is even safer, cpio. It's about as fast as tar, maybe a little quicker.

$ cd $SRC && find . -mount -depth -print0 2>/dev/null | cpio -0admp $DEST &>/dev/null

tar

This is also good, and continues on read-failures.

$ tar --ignore-failed-read -C $SRC -cf - . | tar --ignore-failed-read -C $DEST -xf -

Note those are all just for local copies.

  • Why do you use the -S and -D flags for rsync? – miyalys Jul 14 '15 at 11:37

rsync -aPhW --protocol=28 helps speed up those large copies with RSYNC. I always go rsync because the thought of being midway through 90GiB and it breaking scares me away from CP

  • 2
    What is the value of using the older protocol in that command string? – ewwhite Nov 28 '09 at 3:10
  • 1
    On a mac machine the older version of Rsync shipped hangs on some newer rsync protocol revs such as 29. Telling it to move to the older protocol makes it NOT check over and over again. – oneguynick Jan 3 '10 at 5:52
  • I guess that number 28 is not valid anymore? – SPRBRN Jul 10 '13 at 8:59

The rsync command always computes checksums on every byte it transfers.

The command line option --checksum only relates to whether checksums of files are used to determine which files to transfer or not, ie:

-c, --checksum skip based on checksum, not mod-time & size"

The manpage also says this:

Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking its whole-file checksum, but that automatic after-the-transfer verification has nothing to do with this option’s before-the-transfer "Does this file need to be updated?" check.

So rsync also, always, calculates a checksum of the whole file on the receiving side, even when -c/ --checksum option is "off".

  • 12
    While your post added some interesting information here, the rants, and insults decrease the value of your post. This site is not a forum for un-constructive rants. If you were able to modify the source, have you submitted your modifications as a patch? Have you posted your version on github or something? If you feel so strongly about this, it might be better if you tried to do something a bit more constructive instead of being needlessly insulting. – Zoredache Nov 29 '12 at 21:31
  • Yeah, the last paragraph wasn't really necessary. – Sherwin Flight Oct 25 '15 at 3:57

Whatever you prefer. Just don't forget the -a switch when you decide to use cp.

If you really need an answer: I'd use rsync because it's much more flexible. Need to shutdown before copying is complete? Just ctrl-c and resume as soon as your back. Need to exclude some files? Just use --exclude-from. Need to change ownership or permissions? rsync will do that for you.

  • What does the -p flag do again? – Rory Jul 20 '09 at 15:32
  • 1
    It will Preserver ownership, timestamps and permissions. – innaM Jul 20 '09 at 15:34
  • 5
    cp -a would be better. – David Pashley Jul 20 '09 at 15:36
  • Indeed. Answer changed accordingly. – innaM Jul 20 '09 at 15:53

rsync is great, but has issues with really large directory trees because it stores the trees in memory. I was just looking to see if they'd fix this problem when I found this thread.

I also found:

http://matthew.mceachen.us/geek/gigasync/

You could also manually break up the tree and run multiple rsyncs.

This thread was very useful and because there were so many options to achieve the result, I decided to benchmark few of them. I believe my results can be helpful to others have a sense of what worked faster.

To move 532Gb of data distributed among 1,753,200 files we had those times:

  • rsync took 232 minutes
  • tar took 206 minutes
  • cpio took 225 minutes
  • rsync + parallel took 209 minutes

On my case I preferred to use rsync + parallel. I hope this information helps more people to decide among these alternatives.

The complete benchmark are published here

  • 404 page not found – Amedee Van Gasse Mar 6 at 9:01
  • 1
    Thanks @AmedeeVanGasse URL have been fixed a short after you reported :) – arjones Apr 10 at 1:05
  • Why not benchmarking cp? This is the title of the question! – calandoa Apr 17 at 13:45
  • @calandoa I think cp is insecure, ie: when it breaks you have to start over, that's way I favor options that can resume, ergo rsync is my favorite :) – arjones Apr 19 at 20:07

When doing local a local directory copy, my experience is that "cp -van src dest" is 20% faster than rsync. As far as restartability, that's what "-n" does. You just need to rm the partially copied file. Not painful unless it's an ISO or some such.

ARJ IS SO OLD SCHOOL!! I really doubt that ARJ and/or rsync will give performance.

Definitely what I always do is use cpio:

find . -print | cpio -pdm /target/folder

This is almost fast than CP, definitely faster than tar and without pipeing anything.

  • 2
    "The original cpio and find utilities were written by Dick Haight while working in AT&T's Unix Support Group. They first appeared in 1977 in PWB/UNIX 1.0" - FreeBSD's cpio man page. – Chris S Sep 21 '12 at 2:54
  • 3
    cpio unfortunately has an 8GB upper limit for files. – user139948 Oct 6 '12 at 21:24
  • "without pipeing anything" [sic]. Except the find command, as you listed it, has a pipe in it: find . -print | cpio -pdm /target/folder – warren Oct 7 '15 at 17:15

Both will work just fine.

tar would also do the job, but won't resume from being interrupted like rsync will.

  • An old answer, but isn't TAR for creating compressed archives of files? How could it be used to transfer files like rsync or cp? – Sherwin Flight Oct 25 '15 at 4:01
  • @SherwinFlight cd source ; tar cf - . | ( cd dest ; tar xf - ) – pgs Oct 26 '15 at 6:12

What if you use ARJ?

arj a -jm -m1 -r -je filepack /source

where -jm -m1 are compression levels and -je makes it an executable. Now you have a encapsulated bash of files.

Then for extraction to the target map

filepack -y  

where the source map will be made (where -y is always accept, overwrite, skip etc)

One can then scp ftp the filepack to the target area and execute it, if that is possible.

  • 1
    Arj? Didn't that die out in the 80's? – Michael Hampton Nov 26 '12 at 22:02
  • maybe the early 90's if you believe wikipedia – Matt Nov 28 '12 at 3:22

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.