Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

what is the fastest way to duplicate a folder contain few files with few hundred megs size and other few megs small files? (say, /var/lib/mysql) cp or tar or rsync or...??

share|improve this question

migrated from stackoverflow.com Oct 13 '09 at 10:56

This question came from our site for professional and enthusiast programmers.

    
When you say duplicate, do are you trying to just copy the data to another directory, partition , disk or across a network ? –  Andrew Keith Oct 13 '09 at 8:38
    
the fastest one first (perhaps another directory/partition/disk). then the result will be backup again using other better method accross network. –  DennyHalim.com Oct 13 '09 at 8:42
add comment

7 Answers

up vote 4 down vote accepted

rsync in general will be faster than cp or tar, because rsync only transfers those files which have changed, and with --partial it will only transfer the parts of a file that changed.

Having said that, rsync works much better if you know what you're backupping and can arrange things so that rsync doesn't have to do as much work. For example, rotated log files in /var/log work much better if you rotate them to a filename with a date in it, instead of .0, .1, .2, etc..

One more note: In your question you mention /var/lib/mysql. Using rsync to backup that is a pretty bad idea, use mysqldump to get a reliable backup. If that backup is too large to transfer frequently, use mysql replication and backup from a slave. (You still should do a full backup on the master periodically though, replication can fail too).

share|improve this answer
1  
rom man rsync: "--partial keep partially transferred files". It does not transfer only what changed in a file. If a file changed, the entire file gets transfered. –  churnd Oct 13 '09 at 11:36
1  
Yes, this is slightly inaccurate. --partial keeps partially copied files on the sender if the transfer is aborted. It has nothing to do with the rsync algorithm that transfers only parts of partially changed files, which is on by default (although can be turned off with --whole-file). –  Andrew Ferrier Oct 14 '10 at 16:29
add comment

Depends on a few factors. For a large number of small files it is generally better to do tar. If it is a small number of large files, cp is ok in most instances.

If you have a list of files but only a small portion of them change, rsync is more efficient.

share|improve this answer
    
example is /var/lib/mysql which contain not much files and most of them relative big –  DennyHalim.com Oct 13 '09 at 8:44
1  
If you're planning on backup up mysql files, be sure to have a look at the mysql docs. There are a few traps to watch out for dev.mysql.com/doc/refman/5.0/en/backup-strategy-summary.html –  leonm Oct 13 '09 at 9:15
add comment

It depends on what you are backing up. You can't just backup live database with rsync, you'll end you with a corrupted file. For databases, like mysql, you need to set cron, for example, to run 'mysqldump' and then rsync that file. Also, make sure you append 'date' with the file name so, you know when this file was backed up and have several backup copies just in case.

share|improve this answer
add comment

Ah, the magic word backup

I can't say much about relative speeds of the methods you mention, but I think rsync's ability to backup only files which have changed, makes it the best method.

Especially if most of the files don't need to be copied.

share|improve this answer
add comment

If you have quite a few small files, running a tar over them will increase the transfer speed considerably.

If the files are mostly text (which isn't the case of mysql binaries or databases), consider zipping the tar (bzip2 and the like). Otherwise you won't get anything from the compression so just leave the tar alone.

share|improve this answer
add comment

Assuming that you already have an older copy of the files and you only need to refresh the copy, then rsync is much faster than cp or tar because it only transfers the bytes that have changed. A 100MB file with a few changed records will update in a few seconds. I have found it to be perfect for database files but I have had less success with binary files.

But if the purpose of the copy is for off-site backup, nothing comes near to rdiff-backup. rdiff-backup, like rsync, only transfers changes within files, but it also maintains the incremental changes so that you can restore any file from any earlier backup date.

And, as for the SQL databases, rdiff-backup checks for changes made during the backup and fixes them automatically, so IMHO it should be safe. Unless, of course the SQL server holds data in memory without writing it to disk - then it will not be safe.

share|improve this answer
add comment

SAN Snapshot is the fastest way, but probably not the answer you were looking for.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.