7

I have a large directory called servers, which contains many hard-links made by rsnapshot. That means that the structure is more or less like:

./servers
./servers/daily.0
./servers/daily.0/file1
./servers/daily.0/file2
./servers/daily.0/file3
./servers/daily.1
./servers/daily.1/file1
./servers/daily.1/file2
./servers/daily.1/file3
...

The snapshots were created with rsnapshot in a space-saving way: if /servers/daily.0/file1 is the same as /servers/daily.1/file1, they both point to the same inode using hard-link, instead of just copying a complete snapshot every cycle./servers/daily.0/file1/servers/daily.0/file1

I've tried to copy it with the hard links structure, in order to save space on the destination drive, using:

nohup time rsync -avr --remove-source-files --hard-links servers /old_backups

After some time, the rsync freezes - no new lines are added to the nohup.out file, and no files seem to move from one drive to another. Removing the nohup didn't solve the problem.

Any idea what's wrong?

Adam

4 Answers 4

14

My answer, which I give from hard-earned experience, is: Don't do this. Don't try to copy a directory hierarchy that makes heavy use of hard links, such as one created using rsnapshot or rsync --link-dest or similar. It won't work on anything but small datasets. At least, not reliably. (Your mileage may vary, of course; perhaps your backup datasets are much smaller than mine were.)

The problem with using rsync --hard-links to recreate the hard-linked structure of files on the destination side is that discovering the hard-links on the source side is hard. rsync has to build a map of inodes in memory to find the hard-links, and unless your source has relatively few files, this can and will blow up. In my case, when I learned of this problem and was looking around for alternate solutions, I tried cp -a, which is also supposed to preserve the hard-link structure of files in the destination. It churned away for a long time and then finally died (with a segfault, or something like that).

My recommendation is to set aside an entire partition for your rsnapshot backup. When it fills up, bring another partition online. It is much easier to move around hard-link-heavy datasets as entire partitions, rather than as individual files.

6
  • Plus one to that. You may also want to consider a different approach to the problem, like rdiff-backup, which uses binary diffs instead of hardlinks. (It's got some problems of its own, unfortunately.)
    – mattdm
    Dec 1, 2010 at 3:30
  • 2
    The hardlink program can search for identical files and hardlink them, but it requires them to all have exactly the same attributes (size, contents, permission, owner, group, etc) to work properly. I was able to use it to relink several tens of gigabytes of a music backup in about half an hour.
    – Robbie
    Jul 7, 2012 at 8:10
  • 1
    What do you mean by "blow up"? Exponential slowness? Or does an error occur to let you know that it's not going to work? Jul 22, 2014 at 11:16
  • Even COW file systems (LVM/ZFS/Btrfs) get slow with many copies but they are more robust than hard links. Dec 2, 2019 at 17:10
  • This is excellent information, thank you. I guess I haven't crossed that threshold. My backups are 'incremental' and use -link to create each incremental directory. I actually do 4 an hour then fold them up into daily, then monthly backups. This is done by renaming the directory. My frequent backups have F in their name. Each day I rename the last to D (and remove ofhte F directories) and monthly do the same with D to M, Jun 4, 2023 at 15:42
8

At the point rsync seems to hang, is it hung or just busy? Check for cpu activity with top and disk activity with iotop -o.

It could be busy copying over a large file. You would see this in iotop or similar, or in rsync's display if you ran it with the --progress option.

It could also be busy scanning through lists of inodes to check for linked files. If incremental recursion is being used, which is the default for recursive transfers in most cases if both client and server have rsync v3.0.0 or later, it could have just hit a directory with many files and be running the link check between all the files in it and all those found previously. The --hard-links option can be very CPU intensive over large sets of files (this is why it is not included in the list of options implied by the general --archive option). This will manifest itself as high CPU use at the time rsync seems paused/hung.

2
  • Many people report that it's actually hung. It makes a lot of stat() system calls, bugzilla.samba.org/show_bug.cgi?id=10678#c1 - the difference for the end user whether it's very slow or completely stopped is probably non-existent. Feb 17, 2021 at 18:24
  • 1
    @YaroslavNikitenko - many people incorrectly report that it is actually hung. It has to make all those stat() calls and keep track of the extra information (consuming memory and CPU resource) to do the job it is being asked to do when run over a large tree with those options enabled. Perhaps there could be more progress information while it churns over the required processing, though that wouldn't help in many scripted circumstances when rsync is told to run quiet unless there are actual errors to report. Feb 18, 2021 at 10:47
0

Highly probably rsync is still running but it's going slow enough that you think it has been frozen. Try running sudo iotop -od5 and iostat -tmxyz 5 to show 5 second averages of what's happening in the system.

You could also use top to verify that the rsync is still using CPU power and how much RAM it has acquired. Also check the "Process Status" column. If it says D, the process is waiting for IO (the "D" as in "waiting for Disk access").

In my experience, synchronizing big directories over network is always painful. And if you need to reconstruct hardlinks, doubly so. The hardlinks are especially painful if you already have an old synchronization result and you're trying to update it. For such a case, I'd recommend using flag --delete-before it at all possible (read the side effects first: man rsync). It will compute all the data to be synchronized before moving any data which delays the start of data transmission but allows processing hardlinks efficiently. As a result, it should be able to handle huge amount of hardlinks much better.

Expect to wait at least a couple of hours while rsync is computing the data to be transferred if you use --delete-before and you have lots of files (here lots of files would be 10+ million files spanning over 10+ TB) and RAM usage to be at least a couple of gigabytes (needed to create list of inodes with multiple links if you have lots of hardlinked files). While computing is in progress, very little network traffic is used and if the local and remote systems have highly different performance, the faster system will be idling until the slower system has computed the required changes.

Something like

rsync -avH --delete-before myremoteserver.example.com@/data/. /data/.

should be okay.

And note that if either the local or remote system is running low on RAM, the system may be swapping while trying to process the synchronization at the same time which might result in a state the the rsync process is practically frozen because the whole system is running so slow. Adding enough swap space will probably help in this case.

If you have enough free space on the receiving end, you could simply skip using the -H option to rsync and rebuild the hardlinks to save storage space only later. In that case, you might want to use program called hardlinks.

-1

I had the same problem. My problem was solved by adding --no-inc-recursive option.

From https://download.samba.org/pub/rsync/rsync.html:

If incremental recursion is active (see --recursive), rsync may transfer a missing hard-linked file before it finds that another link for that contents exists elsewhere in the hierarchy.

This does not affect the accuracy of the transfer (i.e. which files are hard-linked together), just its efficiency (i.e. copying the data for a new, early copy of a hard-linked file that could have been found later in the transfer in another member of the hard-linked set of files).

One way to avoid this inefficiency is to disable incremental recursion using the --no-inc-recursive option.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .