29

From time to time, I have to perform several large migration changes on data files on my server, and I'm looking for a good way to do this. I was thinking about using rsync to duplicate my directory structure starting at the root data folder, creating hard links to all original the files (some of them are rather big), and I can overwrite in the destination tree only the files that need migrating. In the end, I can safely switch from the old files to the new files with two mv operations.

However, I can't seem to get rsync to do this. I tried

rsync -a --link-dest=$DATA $DATA $DATA/../upgrade_tmp

but instead of creating hard links to files, rsync copies them entirely. Is there a problem using the same source and link-dest directory?

0

8 Answers 8

27

rsync is a powerful tool, but it is, unfortunately, strangely picky about some of its pathnames.

If $DATA is an absolute path (i.e. it begins with a /), then the correct command-line to use is:

rsync -a --link-dest=$DATA $DATA/ $DATA/../upgrade_tmp

[Now, just a brief aside about rsync's strangeness. Note the trailing / added to the source argument. This tells rsync to work with the contents of the source directory, rather than with the source directory itself. (I'm assuming that $DATA doesn't already contain a trailing /.) In this case, we want to work with the contents, so we add the trailing /.]

If, on the other hand, $DATA is a relative path (i.e. it does not begin with a /), then Sean R's comment about --link-dest is bang on: The link-dest path is interpreted relative to the destination path, so you would use the following:

rsync -a --link-dest=../`basename $DATA` $DATA/ $DATA/../upgrade_tmp

EDIT

One final note, it turns out that the second rsync command-line I gave should work regardless of whether $DATA is an absolute path, since basename doesn't care whether a path is absolute or relative.

3
  • 1
    Just a missing slash, who would have thought of that… Thanks for the nice explanation! Dec 8, 2010 at 9:00
  • Thank you for this, I tried following several incremental backup instructions, like this one and did not find any mention of this quirk. This was the only thing that actually ensured files were being hardlinked. Checking that ref count was > 1 and inode numbers were the same with ls -ilah
    – Walf
    Nov 12, 2013 at 5:11
  • I used this in conjunction with the relpath() function described at unix.stackexchange.com/a/85068/57414 to backup a $SOURCE directory into a $TARGET dir like this: SOURCE='abs_path_to_backup'; TARGET='.'; rsync -a --link-dest=$(relpath $TARGET $SOURCE) $SOURCE/ $TARGET/ Jul 2, 2014 at 3:30
18

What you want is "cp -al":

cp -al $DATA/ $DATA/../upgrade_tmp/
  • -a recurses like rsync -a
  • -l will hard link files instead of copying them.
1
  • 1
    cp -al is unfortunately not available on my system (Mac OS X 10.6). I'll use pax instead... Dec 7, 2010 at 13:08
9

Turns out it is more difficult to do this with rsync than with other tools. The correct answer for rsync is Steven Monai's, but the easiest way to do this is to use either cp -al or pax -rwl on systems where -l is not a valid option for cp:

pax -rwl $DATA $DATA/../upgrade_tmp

or

cp -al $DATA/ $DATA/../upgrade_tmp/
7

The --link-dest option in rsync is relative to the destination directory, not the current directory. So what you want is:

rsync -a --link-dest=../`basename $DATA` $DATA $DATA/../upgrade_tmp
2
  • Oops, I meant basename, originally I had dirname. Dec 7, 2010 at 19:38
  • 2
    The man page says that --link-dest option, if relative, is relative to the target directory. In my case it's absolute. Even making it relative to the target directory does not make this work. Dec 8, 2010 at 8:30
4

It works for me:

$ rsync --hard-links --recursive --link-dest=/local user@host:/remote/ /local

I use rsync version 3.1.0.

From man:

--hard-links

Tells rsync to look for hard-linked files in the transfer, without this option, hard-linked files in the transfer are treated as though they were separate files.

--link-dest=DIR

Unchanged files are hard linked from DIR to the destination directory. The files must be identical in all preserved attributes (e.g. permissions, possibly ownership) in order for the files to be linked together

3
  • 2
    Only code snippet is not enough, explain what it does and why.
    – peterh
    Apr 24, 2015 at 17:39
  • --hard-links Tells rsync to look for hard-linked files in the transfer, without this option, hard-linked files in the transfer are treated as though they were separate files. --link-dest=DIR Unchanged files are hard linked from DIR to the destination directory. The files must be identical in all preserved attributes (e.g. permissions, possibly ownership) in order for the files to be linked together. Apr 30, 2015 at 14:12
  • 1
    Wonderful. Thank you. Actually, I found your answer in the "low quality" queue. It means, there was a vote if your answer should be deleted or not. But not only the danger of the deletions is a reason to try to give a well-formatted, "human" answer, but it also helps a lot if you want to collect a upvotes.
    – peterh
    May 1, 2015 at 2:00
2

Can try following link http://www.lessfs.com/wordpress/ it is work on COW (copy on write) that will save time and space

1
  • lessfs is very interesting, but is also very experimental. Not recommended for production use just yet.
    – mattdm
    Dec 7, 2010 at 17:13
2

First create the directories only on the destination:

rsync -av --include '*/' --exclude '*' /source/ /destination/

Then hard link the files only:

cd /source
find . -type f -exec ln -v {} /destination/{} \;
1
  • Thanks — actually, I might as well use pax as shown in my comment above, which seems easier. Dec 7, 2010 at 12:55
1

Use the Option -H to preserve Hardlinks and read the manpage.

2
  • 1
    -H doesn't work. I have no hardlinks to preserve in my source tree, I just want a simple copy of my source tree where in the copy, all files are hardlinked to the original files. Sorry in my original question was unclear… Dec 7, 2010 at 10:36
  • is "read the manpage" an answer? :-)
    – meduz
    Nov 1, 2015 at 11:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.