Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

From time to time, I have to perform several large migration changes on data files on my server, and I'm looking for a good way to do this. I was thinking about using rsync to duplicate my directory structure starting at the root data folder, creating hard links to all original the files (some of them are rather big), and I can overwrite in the destination tree only the files that need migrating. In the end, I can safely switch from the old files to the new files with two mv operations.

However, I can't seem to get rsync to do this. I tried

rsync -a --link-dest=$DATA $DATA $DATA/../upgrade_tmp

but instead of creating hard links to files, rsync copies them entirely. Is there a problem using the same source and link-dest directory?

share|improve this question
add comment

7 Answers

up vote 9 down vote accepted

rsync is a powerful tool, but it is, unfortunately, strangely picky about some of its pathnames.

If $DATA is an absolute path (i.e. it begins with a /), then the correct command-line to use is:

rsync -a --link-dest=$DATA $DATA/ $DATA/../upgrade_tmp

[Now, just a brief aside about rsync's strangeness. Note the trailing / added to the source argument. This tells rsync to work with the contents of the source directory, rather than with the source directory itself. (I'm assuming that $DATA doesn't already contain a trailing /.) In this case, we want to work with the contents, so we add the trailing /.]

If, on the other hand, $DATA is a relative path (i.e. it does not begin with a /), then Sean R's comment about --link-dest is bang on: The link-dest path is interpreted relative to the destination path, so you would use the following:

rsync -a --link-dest=../`basename $DATA` $DATA/ $DATA/../upgrade_tmp

EDIT

One final note, it turns out that the second rsync command-line I gave should work regardless of whether $DATA is an absolute path, since basename doesn't care whether a path is absolute or relative.

share|improve this answer
    
Just a missing slash, who would have thought of that… Thanks for the nice explanation! –  JPP Dec 8 '10 at 9:00
    
Thank you for this, I tried following several incremental backup instructions, like this one and did not find any mention of this quirk. This was the only thing that actually ensured files were being hardlinked. Checking that ref count was > 1 and inode numbers were the same with ls -ilah –  Walf Nov 12 '13 at 5:11
    
I used this in conjunction with the relpath() function described at unix.stackexchange.com/a/85068/57414 to backup a $SOURCE directory into a $TARGET dir like this: SOURCE='abs_path_to_backup'; TARGET='.'; rsync -a --link-dest=$(relpath $TARGET $SOURCE) $SOURCE/ $TARGET/ –  Nathan S. Watson-Haigh Jul 2 at 3:30
add comment

Can try following link http://www.lessfs.com/wordpress/ it is work on COW (copy on write) that will save time and space

share|improve this answer
    
lessfs is very interesting, but is also very experimental. Not recommended for production use just yet. –  mattdm Dec 7 '10 at 17:13
add comment

Use the Option -H to preserve Hardlinks and read the manpage.

share|improve this answer
1  
-H doesn't work. I have no hardlinks to preserve in my source tree, I just want a simple copy of my source tree where in the copy, all files are hardlinked to the original files. Sorry in my original question was unclear… –  JPP Dec 7 '10 at 10:36
add comment

First create the directories only on the destination:

rsync -av --include '*/' --exclude '*' /source/ /destination/

Then hard link the files only:

cd /source
find . -type f -exec ln -v {} /destination/{} \;
share|improve this answer
    
Thanks — actually, I might as well use pax as shown in my comment above, which seems easier. –  JPP Dec 7 '10 at 12:55
add comment

What you want is "cp -al":

cp -al $DATA/ $DATA/../upgrade_tmp/
  • -a recurses like rsync -a
  • -l will hard link files instead of copying them.
share|improve this answer
    
cp -al is unfortunately not available on my system (Mac OS X 10.6). I'll use pax instead... –  JPP Dec 7 '10 at 13:08
add comment

The --link-dest option in rsync is relative to the destination directory, not the current directory. So what you want is:

rsync -a --link-dest=../`basename $DATA` $DATA $DATA/../upgrade_tmp
share|improve this answer
    
Oops, I meant basename, originally I had dirname. –  Sean Reifschneider Dec 7 '10 at 19:38
    
The man page says that --link-dest option, if relative, is relative to the target directory. In my case it's absolute. Even making it relative to the target directory does not make this work. –  JPP Dec 8 '10 at 8:30
add comment

Turns out it is more difficult to do this with rsync than with other tools. The correct answer for rsync is Steven Monai's, but the easiest way to do this is to use either cp -al or pax -rwl on systems where -l is not a valid option for cp:

pax -rwl $DATA $DATA/../upgrade_tmp

or

cp -al $DATA/ $DATA/../upgrade_tmp/
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.