Highly probably rsync
is still running but it's going slow enough that you think it has been frozen. Try running sudo iotop -od5
and iostat -tmxyz 5
to show 5 second averages of what's happening in the system.
You could also use top
to verify that the rsync is still using CPU power and how much RAM it has acquired. Also check the "Process Status" column. If it says D
, the process is waiting for IO (the "D" as in "waiting for Disk access").
In my experience, synchronizing big directories over network is always painful. And if you need to reconstruct hardlinks, doubly so. The hardlinks are especially painful if you already have an old synchronization result and you're trying to update it. For such a case, I'd recommend using flag --delete-before
it at all possible (read the side effects first: man rsync
). It will compute all the data to be synchronized before moving any data which delays the start of data transmission but allows processing hardlinks efficiently. As a result, it should be able to handle huge amount of hardlinks much better.
Expect to wait at least a couple of hours while rsync
is computing the data to be transferred if you use --delete-before
and you have lots of files (here lots of files would be 10+ million files spanning over 10+ TB) and RAM usage to be at least a couple of gigabytes (needed to create list of inodes with multiple links if you have lots of hardlinked files). While computing is in progress, very little network traffic is used and if the local and remote systems have highly different performance, the faster system will be idling until the slower system has computed the required changes.
Something like
rsync -avH --delete-before myremoteserver.example.com@/data/. /data/.
should be okay.
And note that if either the local or remote system is running low on RAM, the system may be swapping while trying to process the synchronization at the same time which might result in a state the the rsync
process is practically frozen because the whole system is running so slow. Adding enough swap space will probably help in this case.
If you have enough free space on the receiving end, you could simply skip using the -H
option to rsync
and rebuild the hardlinks to save storage space only later. In that case, you might want to use program called hardlinks
.