Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm trying to transfer thousands of small files from one server to another using the following command:

rsync -zr --delete /home/user/ user@10.1.1.1::backup

Currently the transfer takes a long time (I haven't timed it). Is there way to make this faster? Should I be using another tool? Should I be using rsync over ssh rather than using the rsync protocol?

share|improve this question
    
Is it really only hundreds? As in less then a couple thousand? –  Zoredache Mar 1 '12 at 1:41
    
A few more than that... 475,576 totaling 9.3GB –  Noodles Mar 1 '12 at 1:47
    
That is going to suck using almost any tool that works at the filesystem level. I suspect if you did some profiling you would see a significant ammount of time be expended calling stat(). –  Zoredache Mar 1 '12 at 1:58
    
Why not -a but -r? –  kamae Mar 1 '12 at 2:33
add comment

4 Answers

Add -v --progress to your rsync command line

rsync is done in 2 steps:

  1. deep browse all files on both platforms to compare their size and mdate
  2. do the actual transfer

If you are rsync thousands of small files in nested directories, it can simply be that rsync spends most of this time going into subdirs and finding all files

If time is not spend for browsing, the time might simply due to the addition of all the latencies starting each new file transfer.

share|improve this answer
add comment

What version of rsync are you using? Anything older then 3.0.0 (on both ends) doesn't have the incremental filelist feature, which speeds up large transfers.

share|improve this answer
    
Using rsync 3.0.5 on both servers. –  Noodles Mar 1 '12 at 22:40
add comment

You need to determine the bottleneck. It isn't rsync. It probably isn't your network bandwidth. As Zoredache suggested it is most likely the huge number of iops generated by all the stat() calls. Any syncing tool is going to need to stat the files. While syncing run iostat to verify.

So the question becomes: how to I optimize stat? Two easy answers: 1. get a faster disk subsystem (on both hosts if need be) and 2. tune your filesystem (e.g. for ext3 mount with noatime and add a dir_index).

If by some chance it isn't your disk iops that is the limit then you could experiment with splitting the dir tree into multiple distinct trees and run multiple rsyncs.

share|improve this answer
    
Thanks, I'll look into dir_index and see how I get on (we already use noatime). It seems like the disk io is the bottleneck, but we're already running 15k SAS drives in RAID 5. The next step would be SSD, but our hosting company doesn't give us that option yet. –  Noodles Mar 1 '12 at 22:42
add comment

Compression is not very useful for small files (say, less than 100 bytes). For small files, sometimes the compressed version can be even bigger than the original. Try the rsync command without the -z flag.

ssh is good for security, but will not make the transfer faster. In fact, it would make the transfer slower due to the need for encryption/decryption.

rsync may not seem fast the first time it is run because there is a lot of data to transfer. However, if you plan on running this command periodically, subsequent runs may be much faster since rsync is smart about not transferring files that have not changed.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.