Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have to copy a large directory tree, about 1.8 TB. It's all local. Out of habit I'd use rsync, however I wonder if there's much point, and if I should rather use cp.

I'm worried about permissions and uid/gid, since they have to be preserved in the clopy (I know rsync does this). As well as thinks like symlinks.

The destination is empty, so I don't have to worry about conditionally updating some files. It's all local disk access, so I don't have to worry about ssh or network.

The reason I'd be tempted away from rsync, is because rsync might do more than I need. rsync checksums files. I don't need that, and am concerned that it might take longer than cp.

So what do you reckon, rsync or cp?

share|improve this question
add comment

15 Answers 15

up vote 64 down vote accepted

I would use rsync as it means that if it is interrupted for any reason, then you can restart it easily with very little cost. And being rsync, it can even restart part way through a large file. As others mention, it can exclude files easily. The simplest way to preserve most things is to use the -a flag - archive. So:

rsync -a source dest

The default cp will start again, though the -u flag will "copy only when the SOURCE file is newer than the destination file or when the destination file is missing". And the -a (archive) flag will be recursive, not recopy files if you have to restart and preserve permissions. So:

cp -au source dest
share|improve this answer
2  
The -u flag of cp probably isn't the best solution, as it would not detect a partially copied/corrupted file. The nice thing about rsync is that you can have it md5 sum the files to detect differences. –  Chad Huneycutt Jul 20 '09 at 15:10
2  
Adding -w (--whole-file) option would speed up an interrupted rsync, as it will just copy the file over instead of checksumming. –  hayalci Jun 30 '10 at 19:50
4  
actually, rsync detects local transfers and enables whole-file copy without checksumming automagically. –  korkman Oct 8 '12 at 22:49
8  
and --progress which is really handy! –  Matt Nov 28 '12 at 3:20
1  
-P or --progress shows progress for each file individually. It's useful for copying large files, not for many (thousands) small files as it means lots more output which you can't read. It doesn't show the overal progress of all files combined. –  SPRBRN Jul 10 '13 at 8:56
show 1 more comment

When I have to copy a large amount of data, I usually use a combination of tar and rsync. The first pass is to tar it, something like this:

# (cd /src; tar cf - .) | (cd /dst; tar xpf -)

Usually with a large amount of files, there will be some that tar can't handle for whatever reason. Or maybe the process will get interrupted, or if it is a filesystem migration, the you might want to do the initial copy before the actual migration step. At any rate, after the initial copy, I do an rsync step to sync it all up:

# cd /dst; rsync -avPHSx --delete /src/ .

Note that the trailing slash on /src/ is important.

share|improve this answer
2  
+1 I've found tar to generally be faster for large copies than rsync. I like the idea of finishing off with a final rsync, too. –  Geoff Fritz Jul 20 '09 at 16:14
1  
tar is a good choice if the dest dir is empty. Although my way would be: cd $DSTDIR; tar c -C $SRCDIR . | tar –  asdmin Jul 20 '09 at 19:39
    
Doesn't taring the file take up double the space? So if you're transferring 10 GB, you'll need to reserve 20 GB at the destination site? Just checking on this. –  shurane Apr 24 '12 at 17:32
9  
That's the beauty of this method. You do not need double the space because you never actually create an intermediate tar file. The tar before the pipe packs the data and streams it to stdout, and the tar after the pipe grabs it from stdin and unpacks it. –  Chad Huneycutt May 10 '12 at 0:45
1  
I did a cp -a for a 12gb transfer, and this method for a 42gb transfer. The tar method took about 1/4 the time. –  NGaida May 23 at 17:20
add comment

When copying to the local file system I always use the following rsync options:

# rsync -avhW --no-compress --progress /src/ /dst/

Here's my reasoning:

-a is for archive, which preserves ownership, permissions etc.
-v is for verbose, so I can see what's happening (optional)
-h is for human-readable, so the transfer rate and file sizes are easier to read (optional)
-W is for copying whole files only, without delta-xfer algorithm which should reduce CPU load
--no-compress as there's no lack of bandwidth between local devices
--progress so I can see the progress of large files (optional)

I've seen 17% faster transfers using the above rsync settings over the following tar command as suggested by another answer:

# (cd /src; tar cf - .) | (cd /dst; tar xpf -)
share|improve this answer
add comment

rsync

Here is the rsync I use, I prefer cp for simple commands, not this.

$ rsync -ahSD --ignore-errors --force --delete --stats $SRC/ $DIR/

cpio

Here is a way that is even safer, cpio. It's about as fast as tar, maybe a little quicker.

$ cd $SRC && find . -mount -depth -print0 2>/dev/null | cpio -0admp $DEST &>/dev/null

tar

This is also good, and continues on read-failures.

$ tar --ignore-failed-read -C $SRC -cf - . | tar --ignore-failed-read -C $DEST -xf -

Note those are all just for local copies.

share|improve this answer
add comment

rsync -aPhW --protocol=28 helps speed up those large copies with RSYNC. I always go rsync because the thought of being midway through 90GiB and it breaking scares me away from CP

share|improve this answer
1  
What is the value of using the older protocol in that command string? –  ewwhite Nov 28 '09 at 3:10
1  
On a mac machine the older version of Rsync shipped hangs on some newer rsync protocol revs such as 29. Telling it to move to the older protocol makes it NOT check over and over again. –  oneguynick Jan 3 '10 at 5:52
    
I guess that number 28 is not valid anymore? –  SPRBRN Jul 10 '13 at 8:59
add comment

If rsync does exactly what you want it to do, if you are quite familiar with its usage for this particular application already, and if it functions quickly enough to suit your taste, then why on earth would you want to switch?

share|improve this answer
2  
Because I'm concerned that rsync will take longer than cp, since rsync does lots of checksumming that cp won't do –  Rory Jul 20 '09 at 15:31
1  
The cpu overhead of the checksum is small compared to the disk/network i/o. Unless the disk are on the same system and the OS can do some clever drive-drive copy in the bus controller. –  Martin Beckett Jul 20 '09 at 17:07
    
My understanding is that rsync only performs a checksum if you specify the -c flag. Otherwise, it compares file size and modification date. –  richardkmiller Aug 29 '12 at 20:31
    
Checksumming is done on files that differ at the size and timestamp check. If you're paranoid (like after a power outage during copy) you can force checksumming on all files, but on a local transfer, that's usually slower than starting from scratch. –  korkman Oct 8 '12 at 22:53
6  
Maybe he's curious about improving his workflow, and doesn't bury his head in the sand thinking he knows everything. This comment really annoys me. –  Martin Konecny Oct 25 '12 at 1:32
add comment

Whatever you prefer. Just don't forget the -a switch when you decide to use cp.

If you really need an answer: I'd use rsync because it's much more flexible. Need to shutdown before copying is complete? Just ctrl-c and resume as soon as your back. Need to exclude some files? Just use --exclude-from. Need to change ownership or permissions? rsync will do that for you.

share|improve this answer
    
What does the -p flag do again? –  Rory Jul 20 '09 at 15:32
1  
It will Preserver ownership, timestamps and permissions. –  innaM Jul 20 '09 at 15:34
4  
cp -a would be better. –  David Pashley Jul 20 '09 at 15:36
    
Indeed. Answer changed accordingly. –  innaM Jul 20 '09 at 15:53
add comment

rsync is great, but has issues with really large directory trees because it stores the trees in memory. I was just looking to see if they'd fix this problem when I found this thread.

I also found:

http://matthew.mceachen.us/geek/gigasync/

You could also manually break up the tree and run multiple rsyncs.

share|improve this answer
5  
If you use version 3 it doesn't keep the whole tree in memory if it is big, it uses a incremental-recursion algorithm: samba.org/ftp/rsync/src/rsync-3.0.0-NEWS –  Kyle Brandt Jul 20 '09 at 17:09
add comment

Both will work just fine.

share|improve this answer
add comment

When doing local a local directory copy, my experience is that "cp -van src dest" is 20% faster than rsync. As far as restartability, that's what "-n" does. You just need to rm the partially copied file. Not painful unless it's an ISO or some such.

share|improve this answer
add comment

I used the suggestion from oneguynick rsync -vaPhW --protocol=28 with a added v for verbose to get some status information during the process.

share|improve this answer
add comment

People are incorrect to state that rsync detects local file transfers and avoids checksumming.

rsync ALWAYS checksums EVERY BYTE it transfers.

The checksumming command line options only relate to whether checksums of files are used to determine which files to transfer or not, ie:

"-c, --checksum skip based on checksum, not mod-time & size"

What is worse, is the manpage also says this:

"Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking its whole-file checksum, but that automatic after-the-transfer verification has nothing to do with this option’s before-the-transfer "Does this file need to be updated?" check."

So rsync ALSO, ALWAYS, calculates a checksum of the whole file on the receiving side, even when -c/ --checksum option is OFF.

We have Dell R510/R710 servers here, plus our own custom built servers with fast RAID cards, and, for example, on a Dell R710 with dual 6-core CPUs, hyperthreading on, ie 24 cores - it is pitiful to see rsync bottlenecking our copies due to CPU load on several cores.

People forget that while CPUs and cores are multiplying like flies (12-core or 24-core machines are common now) CPUs are not, individually getting much faster. So it is easy for apps which are single-threaded or have a small number of threads to choke on CPU, while 20 cores are sitting idle and the 10-disk raid array plugs away at 1/10th of its maximum speed.

I got so fed up with this I went and haxxored the rsync sources to REALLY disable the checksumming on local file copies, and then it was blindingly fast. But the checksumming code is embeded deeply into rsync and the way I disabled it was an ugly hack that was not safe to use generally. So I've kept this ugly hack to myself.

Someone with a little bit more time should FIX RSYNC PROPERLY because its an unbelievable joke that rsync behaves this way on local file copies. The jokers who maintain the rsync code are halfwits who have been asked to disable the checksumming on local copies, but they are too thick to understand how much this cripples rsync. They seriously think rsync is only used for copies over networks etc, where such checksumming is important. They're too thick to see rsync has many advantages over cp even for local copies. In fact, cp is unusable, in many situations, and rsync is the only option. But this local checksuming cripples it.

share|improve this answer
6  
While your post added some interesting information here, the rants, and insults decrease the value of your post. This site is not a forum for un-constructive rants. If you were able to modify the source, have you submitted your modifications as a patch? Have you posted your version on github or something? If you feel so strongly about this, it might be better if you tried to do something a bit more constructive instead of being needlessly insulting. –  Zoredache Nov 29 '12 at 21:31
add comment

ARJ IS SO OLD SCHOOL!! I really doubt that ARJ and/or rsync will give performance.

Definitely what I always do is use cpio:

find . -print | cpio -pdm /target/folder

This is almost fast than CP, definitely faster than tar and without pipeing anything.

share|improve this answer
2  
"The original cpio and find utilities were written by Dick Haight while working in AT&T's Unix Support Group. They first appeared in 1977 in PWB/UNIX 1.0" - FreeBSD's cpio man page. –  Chris S Sep 21 '12 at 2:54
1  
cpio unfortunately has an 8GB upper limit for files. –  user139948 Oct 6 '12 at 21:24
add comment

tar would also do the job, but won't resume from being interrupted like rsync will.

share|improve this answer
add comment

What if you use ARJ?

arj a -jm -m1 -r -je filepack /source

where -jm -m1 are compression levels and -je makes it an executable. Now you have a encapsulated bash of files.

Then for extraction to the target map

filepack -y  

where the source map will be made (where -y is always accept, overwrite, skip etc)

One can then scp ftp the filepack to the target area and execute it, if that is possible.

share|improve this answer
    
Arj? Didn't that die out in the 80's? –  Michael Hampton Nov 26 '12 at 22:02
    
maybe the early 90's if you believe wikipedia –  Matt Nov 28 '12 at 3:22
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.