rsync is just hanging, building file list

Question

[root@centos /]# rsync -av --exclude thumbs /storage root@xx.27.1.xx:/storage root@xx.27.1.xx's password: building file list ...

been sitting for about an hour... it's 135GB of images and folders

/storage is a mounted ext3 scsi drive.

is it normal for rsync to be sitting this long calculating the files/directories?

Answer 1

rsync 2.x does build a full file list up front.

Add the -P option and you'll see a progress indicator.

If you think it really is hanging, in another terminal find the pid of rsync and then

1. Run strace -p PID and see what it's doing. (Hit ^C to stop.)

2. Run ps -o wchan PID to see where it is in the kernel.

Answer 2

You should upgrade to rsync 3.0.x where you will get the benefit of incremental file lists, explained here. I am rsyncing millions of images (~200GB total) and saw an enormous speedup when going from rsync 2.x to 3.x.

Even so, it will probably take a long time to go through all that data. In my case it still takes over an hour between two pretty powerful DL380 G5 servers.

Answer 3

You noted earlier that du was hanging as well. Something is up with this filesystem, and rsync is hitting whatever du was.

One quick thing you can check is for kernel error messages indicating a disk problem. Type dmesg and see what's at the bottom.

Answer 4

What version of rsync are you using? Older versions had memory issues with large numbers of files.

Answer 5

version 3 of rsync is much better with a ton of files

Answer 6

what O/S ? can you use lsof or strace/truss/tusc to see what the rsync process is doing ?

Answer 7

Based on this and your other question of du /storage hanging makes me think that neither du or rsync is the problem but rather there are some issues with /storage.

Answer 8

every Friday o rsync 2.5 million files, in a 1.5 TB array, it takes less than an hour (including the copy of the new/changed files) rsync 3.x