24 
[root@centos /]# rsync -av --exclude thumbs /storage [email protected]:/storage
[email protected]'s password: 
building file list ...

I've been sitting for about an hour... it's 135GB of images and folders

/storage is a mounted ext3 scsi drive.

Is it normal for rsync to be sitting this long calculating the files/directories?

Improve this question
6
  • How many files? With very large numbers of files it's gonna take a while.
    – Helvick
    Dec 2, 2010 at 19:24
  • probably thousands... images directory of user uploads...
    – Andrew Fashion
    Dec 2, 2010 at 19:37
  • I have seen this portion of rsync take well over an hour when there's thousands and thousands and thousands of files. If you run 'top' are you seeing I/O wait?
    – user15590
    Dec 2, 2010 at 19:52
  • Just because I am curious, can you run df -i and tell us what you have for IUsed on /storage? It will give us a rough idea of how many files+folders you have.
    – Zoredache
    Dec 2, 2010 at 21:06
  • 1383641, is that the amount of files?
    – Andrew Fashion
    Dec 2, 2010 at 21:24

5 Answers 5

Reset to default
28

rsync 2.x does build a full file list up front.

Add the -P option and you'll see a progress indicator.

If you think it really is hanging, in another terminal find the pid of rsync and then

  1. Run strace -p PID and see what it's doing. (Hit ^C to stop.)

  2. Run ps -o wchan PID to see where it is in the kernel.

Improve this answer
4
  • 1
    Wow strace is cool! You'll probably find that something circular is going on: Like a link that refers to one of its ancestor directories. In my case I was using a virtual filesystem that had no limit on its depth, i.e. it had "infinitely" many subdirectories tagfs/books/+/books/+/books/+/ ...
    – Zaz
    May 17, 2015 at 16:28
  • @Zaz I also have a huge amount of subdirectories, in depth. But this is in my system design. Is there a way to solve this?
    – user1641443
    Sep 14, 2016 at 11:56
  • @user1641443: I'm sorry, that's beyond me.
    – Zaz
    Sep 16, 2016 at 7:30
  • @user1641443: 1: Try rsync>3.0 as Martin says. 2: Run multiple rsync processes each on a on smaller sub-tree. 3: Open your own separate question.
    – poolie
    Sep 17, 2016 at 15:25
16

You should upgrade to rsync 3.0.x where you will get the benefit of incremental file lists, explained here. I am rsyncing millions of images (~200GB total) and saw an enormous speedup when going from rsync 2.x to 3.x.

Even so, it will probably take a long time to go through all that data. In my case it still takes over an hour between two pretty powerful DL380 G5 servers.

Improve this answer
2
  • 1
    Also note that certain flags will force the old behavior; for me, it was --delay-updates.
    – Xiong Chiamiov
    Jun 6, 2016 at 22:02
  • 1
    @XiongChiamiov The only options I'm using are "-a" and "--progress" and version 3.1.x is still building an incremental file list. Why?
    – Michael
    Apr 23, 2017 at 22:35
3

You noted earlier that du was hanging as well. Something is up with this filesystem, and rsync is hitting whatever du was.

One quick thing you can check is for kernel error messages indicating a disk problem. Type dmesg and see what's at the bottom.

Improve this answer
1

Based on this and your other question of du /storage hanging makes me think that neither du or rsync is the problem but rather there are some issues with /storage.

Improve this answer
1

The hang after the message "building file list..." can be caused by MTU mismatch, e.g. if you've set up the network interfaces with MTU 9000 but not the switch in between, small packets (like connecting to the rsync server) will pass through but not bigger ones (like sending the file list).

Improve this answer
1
  • after much searching I came across this and it prompted me to go and check - was running rsync over a vpn with standard 1450 mtu but I am not in control of the equipment in between the sites - changed the vpn mtu down to 1430 and viola the rsync works 100% (and suddenly other network strangeness disappeared too!) thanks!
    – l0ft13
    Nov 10, 2018 at 10:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .