up vote 22 down vote favorite
13
share [g+] share [fb]

I have to copy a large directory tree, about 1.8 TB. It's all local. Out of habit I'd use rsync, however I wonder if there's much point, and if I should rather use cp.

I'm worried about permissions and uid/gid, since they have to be preserved in the clopy (I know rsync does this). As well as thinks like symlinks.

The destination is empty, so I don't have to worry about conditionally updating some files. It's all local disk access, so I don't have to worry about ssh or network.

The reason I'd be tempted away from rsync, is because rsync might do more than I need. rsync checksums files. I don't need that, and am concerned that it might take longer than cp.

So what do you reckon, rsync or cp?

link|improve this question
79% accept rate
feedback

11 Answers

up vote 19 down vote accepted

I would use rsync as it means that if it is interrupted for any reason, then you can restart it easily with very little cost. And being rsync, it can even restart part way through a large file. As others mention, it can exclude files easily. The simplest way to preserve most things is to use the -a flag - archive. So:

rsync -a source dest

The default cp will start again, though the -u flag will "copy only when the SOURCE file is newer than the destination file or when the destination file is missing". And the -a (archive) flag will be recursive, not recopy files if you have to restart and preserve permissions. So:

cp -au source dest
link|improve this answer
The -u flag of cp probably isn't the best solution, as it would not detect a partially copied/corrupted file. The nice thing about rsync is that you can have it md5 sum the files to detect differences. – Chad Huneycutt Jul 20 '09 at 15:10
1  
Adding -w (--whole-file) option would speed up an interrupted rsync, as it will just copy the file over instead of checksumming. – hayalci Jun 30 '10 at 19:50
feedback
up vote 16 down vote

When I have to copy a large amount of data, I usually use a combination of tar and rsync. The first pass is to tar it, something like this:

# (cd /src; tar cf - .) | (cd /dst; tar xpf -)

Usually with a large amount of files, there will be some that tar can't handle for whatever reason. Or maybe the process will get interrupted, or if it is a filesystem migration, the you might want to do the initial copy before the actual migration step. At any rate, after the initial copy, I do an rsync step to sync it all up:

# cd /dst; rsync -avPHSx --delete /src/ .

Note that the trailing slash on /src/ is important.

link|improve this answer
2  
+1 I've found tar to generally be faster for large copies than rsync. I like the idea of finishing off with a final rsync, too. – Geoff Fritz Jul 20 '09 at 16:14
tar is a good choice if the dest dir is empty. Although my way would be: cd $DSTDIR; tar c -C $SRCDIR . | tar – asdmin Jul 20 '09 at 19:39
Doesn't taring the file take up double the space? So if you're transferring 10 GB, you'll need to reserve 20 GB at the destination site? Just checking on this. – shurane Apr 24 at 17:32
That's the beauty of this method. You do not need double the space because you never actually create an intermediate tar file. The tar before the pipe packs the data and streams it to stdout, and the tar after the pipe grabs it from stdin and unpacks it. – Chad Huneycutt May 10 at 0:45
feedback
up vote 6 down vote

If rsync does exactly what you want it to do, if you are quite familiar with its usage for this particular application already, and if it functions quickly enough to suit your taste, then why on earth would you want to switch?

link|improve this answer
Because I'm concerned that rsync will take longer than cp, since rsync does lots of checksumming that cp won't do – Rory Jul 20 '09 at 15:31
1  
The cpu overhead of the checksum is small compared to the disk/network i/o. Unless the disk are on the same system and the OS can do some clever drive-drive copy in the bus controller. – Martin Beckett Jul 20 '09 at 17:07
feedback
up vote 5 down vote

rsync -aPhW --protocol=28 helps speed up those large copies with RSYNC. I always go rsync because the thought of being midway through 90GiB and it breaking scares me away from CP

link|improve this answer
1  
What is the value of using the older protocol in that command string? – ewwhite Nov 28 '09 at 3:10
On a mac machine the older version of Rsync shipped hangs on some newer rsync protocol revs such as 29. Telling it to move to the older protocol makes it NOT check over and over again. – oneguynick Jan 3 '10 at 5:52
feedback
up vote 3 down vote

Whatever you prefer. Just don't forget the -a switch when you decide to use cp.

If you really need an answer: I'd use rsync because it's much more flexible. Need to shutdown before copying is complete? Just ctrl-c and resume as soon as your back. Need to exclude some files? Just use --exclude-from. Need to change ownership or permissions? rsync will do that for you.

link|improve this answer
What does the -p flag do again? – Rory Jul 20 '09 at 15:32
1  
It will Preserver ownership, timestamps and permissions. – innaM Jul 20 '09 at 15:34
3  
cp -a would be better. – David Pashley Jul 20 '09 at 15:36
Indeed. Answer changed accordingly. – innaM Jul 20 '09 at 15:53
feedback
up vote 3 down vote

rsync is great, but has issues with really large directory trees because it stores the trees in memory. I was just looking to see if they'd fix this problem when I found this thread.

I also found:

http://matthew.mceachen.us/geek/gigasync/

You could also manually break up the tree and run multiple rsyncs.

link|improve this answer
3  
If you use version 3 it doesn't keep the whole tree in memory if it is big, it uses a incremental-recursion algorithm: samba.org/ftp/rsync/src/rsync-3.0.0-NEWS – Kyle Brandt♦ Jul 20 '09 at 17:09
feedback
up vote 1 down vote

Both will work just fine.

link|improve this answer
feedback
up vote 1 down vote

When doing local a local directory copy, my experience is that "cp -van src dest" is 20% faster than rsync. As far as restartability, that's what "-n" does. You just need to rm the partially copied file. Not painful unless it's an ISO or some such.

link|improve this answer
feedback
up vote 1 down vote

rsync

Here is the rsync I use, I prefer cp for simple commands, not this.

$ rsync -ahSD --ignore-errors --force --delete --stats $SRC/ $DIR/

cpio

Here is a way that is even safer, cpio. It's about as fast as tar, maybe a little quicker.

$ cd $SRC && find . -mount -depth -print0 2>/dev/null | cpio -0admp $DEST &>/dev/null

tar

This is also good, and continues on read-failures.

$ tar --ignore-failed-read -C $SRC -cf - . | tar --ignore-failed-read -C $DEST -xf -

Note those are all just for local copies.

link|improve this answer
feedback
up vote 0 down vote

tar would also do the job, but won't resume from being interrupted like rsync will.

link|improve this answer
feedback
up vote 0 down vote

What if you use ARJ

arj a -jm -m1 -r -je filepack /source

where -jm -m1 are compression levels and -je makes it an executable

now you have a encapsulated bash of files

extraction to the target map

filepack -y
where the source map will be made (where -y is always accept, overwrite, skip etc)

one can scp ftp the filepack to the target area and execute it if that is possible

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.