Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

[root@centos /]# rsync -av --exclude thumbs /storage root@xx.27.1.xx:/storage root@xx.27.1.xx's password: building file list ...

been sitting for about an hour... it's 135GB of images and folders

/storage is a mounted ext3 scsi drive.

is it normal for rsync to be sitting this long calculating the files/directories?

share|improve this question
    
How many files? With very large numbers of files it's gonna take a while. –  Helvick Dec 2 '10 at 19:24
    
probably thousands... images directory of user uploads... –  Andrew Fashion Dec 2 '10 at 19:37
    
I have seen this portion of rsync take well over an hour when there's thousands and thousands and thousands of files. If you run 'top' are you seeing I/O wait? –  troyengel Dec 2 '10 at 19:52
    
Just because I am curious, can you run df -i and tell us what you have for IUsed on /storage? It will give us a rough idea of how many files+folders you have. –  Zoredache Dec 2 '10 at 21:06
    
1383641, is that the amount of files? –  Andrew Fashion Dec 2 '10 at 21:24
show 1 more comment

4 Answers

rsync 2.x does build a full file list up front.

Add the -P option and you'll see a progress indicator.

If you think it really is hanging, in another terminal find the pid of rsync and then

  1. Run strace -p PID and see what it's doing. (Hit ^C to stop.)

  2. Run ps -o wchan PID to see where it is in the kernel.

share|improve this answer
add comment

You should upgrade to rsync 3.0.x where you will get the benefit of incremental file lists, explained here. I am rsyncing millions of images (~200GB total) and saw an enormous speedup when going from rsync 2.x to 3.x.

Even so, it will probably take a long time to go through all that data. In my case it still takes over an hour between two pretty powerful DL380 G5 servers.

share|improve this answer
add comment

You noted earlier that du was hanging as well. Something is up with this filesystem, and rsync is hitting whatever du was.

One quick thing you can check is for kernel error messages indicating a disk problem. Type dmesg and see what's at the bottom.

share|improve this answer
add comment

Based on this and your other question of du /storage hanging makes me think that neither du or rsync is the problem but rather there are some issues with /storage.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.