Copying a large directory tree locally? cp or rsync?

Question

I have to copy a large directory tree, about 1.8 TB. It's all local. Out of habit I'd use rsync, however I wonder if there's much point, and if I should rather use cp.

I'm worried about permissions and uid/gid, since they have to be preserved in the clopy (I know rsync does this). As well as thinks like symlinks.

The destination is empty, so I don't have to worry about conditionally updating some files. It's all local disk access, so I don't have to worry about ssh or network.

The reason I'd be tempted away from rsync, is because rsync might do more than I need. rsync checksums files. I don't need that, and am concerned that it might take longer than cp.

So what do you reckon, rsync or cp?

Answer 1

I would use rsync as it means that if it is interrupted for any reason, then you can restart it easily with very little cost. And being rsync, it can even restart part way through a large file. As others mention, it can exclude files easily. The simplest way to preserve most things is to use the -a flag - archive. So:

rsync -a source dest

The default cp will start again, though the -u flag will "copy only when the SOURCE file is newer than the destination file or when the destination file is missing". And the -a (archive) flag will be recursive, not recopy files if you have to restart and preserve permissions. So:

cp -au source dest

Answer 2

When I have to copy a large amount of data, I usually use a combination of tar and rsync. The first pass is to tar it, something like this:

# (cd /src; tar cf - .) | (cd /dst; tar xpf -)

Usually with a large amount of files, there will be some that tar can't handle for whatever reason. Or maybe the process will get interrupted, or if it is a filesystem migration, the you might want to do the initial copy before the actual migration step. At any rate, after the initial copy, I do an rsync step to sync it all up:

# cd /dst; rsync -avPHSx --delete /src/ .

Note that the trailing slash on /src/ is important.

Answer 3

rsync

Here is the rsync I use, I prefer cp for simple commands, not this.

$ rsync -ahSD --ignore-errors --force --delete --stats $SRC/ $DIR/

cpio

Here is a way that is even safer, cpio. It's about as fast as tar, maybe a little quicker.

$ cd $SRC && find . -mount -depth -print0 2>/dev/null | cpio -0admp $DEST &>/dev/null

tar

This is also good, and continues on read-failures.

$ tar --ignore-failed-read -C $SRC -cf - . | tar --ignore-failed-read -C $DEST -xf -

Note those are all just for local copies.

Answer 4

If rsync does exactly what you want it to do, if you are quite familiar with its usage for this particular application already, and if it functions quickly enough to suit your taste, then why on earth would you want to switch?

Answer 5

rsync -aPhW --protocol=28 helps speed up those large copies with RSYNC. I always go rsync because the thought of being midway through 90GiB and it breaking scares me away from CP

Answer 6

Whatever you prefer. Just don't forget the -a switch when you decide to use cp.

If you really need an answer: I'd use rsync because it's much more flexible. Need to shutdown before copying is complete? Just ctrl-c and resume as soon as your back. Need to exclude some files? Just use --exclude-from. Need to change ownership or permissions? rsync will do that for you.

Answer 7

rsync is great, but has issues with really large directory trees because it stores the trees in memory. I was just looking to see if they'd fix this problem when I found this thread.

I also found:

http://matthew.mceachen.us/geek/gigasync/

You could also manually break up the tree and run multiple rsyncs.

Answer 8

When copying to the local file system I always use the following rsync options:

# rsync -avhW --no-compress --progress /src/ /dst/

Here's my reasoning:

-a is for archive, which preserves ownership, permissions etc.
-v is for verbose, so I can see what's happening (optional)
-h is for human-readable, so the transfer rate and file sizes are easier to read (optional)
-W is for copying whole files only, without delta-xfer algorithm which should reduce CPU load
--no-compress as there's no lack of bandwidth between local devices
--progress so I can see the progress of large files (optional)

I've seen 17% faster transfers using the above rsync settings over the following tar command as suggested by another answer:

# (cd /src; tar cf - .) | (cd /dst; tar xpf -)

Answer 9

Both will work just fine.

Answer 10

When doing local a local directory copy, my experience is that "cp -van src dest" is 20% faster than rsync. As far as restartability, that's what "-n" does. You just need to rm the partially copied file. Not painful unless it's an ISO or some such.

Answer 11

I used the suggestion from oneguynick rsync -vaPhW --protocol=28 with a added v for verbose to get some status information during the process.

Answer 12

People are incorrect to state that rsync detects local file transfers and avoids checksumming.

rsync ALWAYS checksums EVERY BYTE it transfers.

The checksumming command line options only relate to whether checksums of files are used to determine which files to transfer or not, ie:

"-c, --checksum skip based on checksum, not mod-time & size"

What is worse, is the manpage also says this:

"Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking its whole-file checksum, but that automatic after-the-transfer verification has nothing to do with this option’s before-the-transfer "Does this file need to be updated?" check."

So rsync ALSO, ALWAYS, calculates a checksum of the whole file on the receiving side, even when -c/ --checksum option is OFF.

We have Dell R510/R710 servers here, plus our own custom built servers with fast RAID cards, and, for example, on a Dell R710 with dual 6-core CPUs, hyperthreading on, ie 24 cores - it is pitiful to see rsync bottlenecking our copies due to CPU load on several cores.

People forget that while CPUs and cores are multiplying like flies (12-core or 24-core machines are common now) CPUs are not, individually getting much faster. So it is easy for apps which are single-threaded or have a small number of threads to choke on CPU, while 20 cores are sitting idle and the 10-disk raid array plugs away at 1/10th of its maximum speed.

I got so fed up with this I went and haxxored the rsync sources to REALLY disable the checksumming on local file copies, and then it was blindingly fast. But the checksumming code is embeded deeply into rsync and the way I disabled it was an ugly hack that was not safe to use generally. So I've kept this ugly hack to myself.

Someone with a little bit more time should FIX RSYNC PROPERLY because its an unbelievable joke that rsync behaves this way on local file copies. The jokers who maintain the rsync code are halfwits who have been asked to disable the checksumming on local copies, but they are too thick to understand how much this cripples rsync. They seriously think rsync is only used for copies over networks etc, where such checksumming is important. They're too thick to see rsync has many advantages over cp even for local copies. In fact, cp is unusable, in many situations, and rsync is the only option. But this local checksuming cripples it.

Answer 13

ARJ IS SO OLD SCHOOL!! I really doubt that ARJ and/or rsync will give performance.

Definitely what I always do is use cpio:

find . -print | cpio -pdm /target/folder

This is almost fast than CP, definitely faster than tar and without pipeing anything.

Answer 14

tar would also do the job, but won't resume from being interrupted like rsync will.

Answer 15

What if you use ARJ?

arj a -jm -m1 -r -je filepack /source

where -jm -m1 are compression levels and -je makes it an executable. Now you have a encapsulated bash of files.

Then for extraction to the target map

filepack -y  

where the source map will be made (where -y is always accept, overwrite, skip etc)

One can then scp ftp the filepack to the target area and execute it, if that is possible.