Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Basically I am searching to see if there exists a tool or script that can detect moved or renamed files so that I can get a list of renamed/moved files and apply the same operation on the other end of the network to conserve on bandwidth.

Basically disk storage is cheap but bandwidth isn't, and the problem is that the files often will be reorganized or moved around into a better directory structure thus when you use rsync to do the backup, rsync won't notice that its a renamed or moved file and re-transmission it over the network all over again despite having the same file on the other end.

So I am wondering if there exists a script or tool that can record where all the files are and their names, then just prior to a backup, it would rescan and detect moved or renamed files, then I can take that list and re-apply the move/rename operation on the other side.

Here's a list of the "general" features of the files:

  1. Large unchanging files
  2. They can be renamed or moved around

[Edit:] These all are good answers, and what I end up doing in the end was looking at all of the answers and will be writing some code to deal with this. Basically what I am thinking/working on now is:

  1. Using something like AIDE for the "initial" scan and enable me to keep checksums on the files because they are supposed to never change, so it would aid on detecting corruption.
  2. Creating an inotify daemon that would monitor these files/directory and recording any changes relating to renames & moving the files around to a log file.
  3. There are some edge cases where inotify might fail to record that something happened to the file system, thus there is a final step of using find to search the file system for files that has a change time latter than the last backup.

This has several benefits:

  1. Checksums/etc from AIDE to be able to check/make sure that some media did not get corrupt
  2. Inotify keeps resource usage low and no need to re-scan the filesystem over and over
  3. No need to patch rsync; If I have to patch things I can, but I would prefer to avoid patching things to keep the burden lower, (IE don't need to re-patch everytime there is an update).
  4. I've used Unison before and its really nice, however I could've sworn that Unison does keep copies around on the filesystem and that its "archive" files can grow to be rather large?
share|improve this question
1  
Two years later... What solution did you end up with? Would you post that as an answer and accept it? Or is this question actually still unsolved? –  Tobias Kienzler Oct 4 '12 at 8:18
add comment

9 Answers

Unison http://www.cis.upenn.edu/~bcpierce/unison/ claims to be able to detect moves and renames.

There are a couple patches to rsync to add move/rename detection:

http://gitweb.samba.org/?p=rsync-patches.git;a=blob;f=detect-renamed-lax.diff;h=1ff593c8f97a97e8970d43ff5a62dfad5abddd75;hb=master

http://gitweb.samba.org/?p=rsync-patches.git;a=blob;f=detect-renamed.diff;h=c3e6e846eab437e56e25e2c334e292996ee84345;hb=master

Bugzilla entry tracking this issue: https://bugzilla.samba.org/show_bug.cgi?id=2294

share|improve this answer
6  
Why aren't these patches integrated? They just add flags, they're not intrusive. Another interesting patch is rsyncsums, which can keep checksums around between rsync runs. –  Tobu Nov 12 '10 at 21:44
add comment

This is a bit of an odd solution, but... git detects moves and renames based on file content, so if you were to keep the directories in question under version control then git would be able to detect moves and such and avoid transferring the content (since it's already on both sides of the wire) while still moving things around in the tree.

Just a thought.

share|improve this answer
1  
Yes I considered this, if the files were small and text based, this would probably work good, but they are binary and the total size is approaching a Terabyte. –  Pharaun Aug 18 '10 at 16:21
    
@Pharaun You'd need the git index without the blob storage. Maybe rip this code out of git and add it to libgit2. –  Tobu Nov 12 '10 at 21:58
    
The relevant code starts with refresh_index in read-cache.c . –  Tobu Nov 12 '10 at 22:03
add comment

You might be able to use a host based IDS such as AIDE and write a wrapper script using its output. You would likely have to write more complex logic considering the checksums.

Otherwise, a network based filesystem might make sense, as the changes would be reflected at all locations. Nevertheless, I suspect you are transferring over the Internet, which will limit options here.

share|improve this answer
    
That was what I was thinking of doing, taking one of those and extend them. Also yes I am transferring it over the internet and the bandwidth is pretty limited. –  Pharaun Aug 18 '10 at 15:41
add comment

interesting suggestions here. Also thought of using filesystem capabilities ie ZFS. Found it strange that there is no tool which does that simple thing. Unison option does not work in most cases as people report, not for me either.

I want the feature to keep backup of my movie collection on second hard disk in sync when rearraring folders.

Now i found this simple C script http://sourceforge.net/projects/movesync/

Seems to work fine. Run it and then sync normally with ie unison.

share|improve this answer
add comment

You might try unison ; especially the

-xferbycopying optimize transfers using local copies (default true)

option mentioned in the docs as

When this preference is set, Unison will try to avoid transferring file contents across the network by recognizing when a file with the required contents already exists in the target replica. This usually allows file moves to be propagated very quickly. The default value is true.

looks like it might do what you want.

share|improve this answer
    
Actually in hindsight, I might have been too hasty on the unison comment. Does unison support replacing a hardlink with the actual file content if it changes? If so then I might be able to do some magic with rsnapshot + unison which would meet my requirements without having to write up a ton of new code/log/etc to deal with this. –  Pharaun Aug 23 '10 at 15:42
add comment

Syrep does what you need. It keeps message digests on a file tree up to date; keeping the digests around makes it more efficient than rsync. It was designed for sneakernet so you may want to add a wrapper that does update/makepatch/merge at once.

share|improve this answer
add comment

I'm not sure if there's an existing tool that does this for you, but you could write a simple script that just runs a find on the base directory where mtime is newer than the last backup. This will get you a list of all files that have been modified. If a file was simply moved, it will not appear in the list. Unfortunately, this list will include the directories that the files moved into, since the directory gets updated when a file is added/removed.

With that list of files, you can use rsync to only sync those files. rsync has an option to read in a file list. Here's a test showing this example:

$ cd tmp
$ echo test > test
$ ls -la
total 16
drwxr-xr-x 2 root root 4096 Aug 18 11:34 .
drwxr-x--- 5 root root 4096 Aug 18 11:34 ..
-rw-r--r-- 1 root root    5 Aug 18 11:34 test
$ mkdir tmp2
$ find . -mmin 1
$ date
Wed Aug 18 11:35:10 EDT 2010
$ find . -mmin 1
$ find . -mmin 2
.
./test
./tmp2
$ mv test tmp2
$ find . -mmin 1
.
./tmp2

Please note that I waited approximately 1 minute between running each find command. From this, it shows that when initially creating the file, it gets listed by find. If I move the file into another directory and re-run the find command, it only displays the directory I moved the file into, and not the file itself. You can use a combination of find and rsync commands to only list the files you want, it can probably achieve your goal.

I hope this helps.

share|improve this answer
add comment

Given your workflow, I wonder if working at the file level (like what others have proposed so far) is the best solution. You could work...

At the filesystem level

The idea is to have the filesystem keep track of operations between backups. Instead of making a backup of the filesystem, back up the filesystem journal (and optionally replay the changes on the backup machine, if you want a ready-to-use backup). A filesystem journal naturally expresses moves and deletions in a few bytes.

Fuse makes it relatively easy to design a filesystem with specific requirements that sits on top of a “real filesystem”. I've never used it, but LoggedFS looks promising.

With this solution, it would be worthwhile to have some form of journal compression. For example, if a file has been overwritten 10 times, only keep its last update in the journal. Another worthwhile optimization would be to recognize copy operations, and even better, edits (i.e., creating a file that is mostly but not completely identical to another file). I don't know if anybody has implemented this. For your workflow, I don't think it would matter much anyway.

At the volume level

The idea is to have the volume manager keep track of operations between backups. Instead of making a backup of the filesystem, take a snapshot with the volume manager and back up the snapshot expressed as a diff from the previous snapshot.

This should work well if all you do is create files, rename them and remove them. It would be a lot harder to detect things like copies and edits, or to optimize away the creation of a file followed by its deletion.

share|improve this answer
    
I've actually been working a bit on a file "system" logger via inotify to keep track of changes, but if the changes comes in faster than the speed that the daemon can record it, it will lose information, hence need to build a backup/scan to get the initial state and in case of inotify losing information. It does look like the idea of having something that sits between the filesystem & rest of the system might also be a good idea then like you said, that changes could be replayed on the backup machine. –  Pharaun Aug 24 '10 at 13:45
    
But that loggedFS does look like an interesting project, only concern is they stopped dev in 2008/09. Going to have to play with it and see if it will do the trick. –  Pharaun Aug 24 '10 at 13:48
add comment
"Basically I am searching to see if there exists a tool or script that can detect moved or renamed files so that I can get a list of renamed/moved files and apply the same operation on the other end of the network to conserve on bandwidth."

How is it that files are moving about? Is this moving of files something you have control over?

If you do, then maybe your solution is in the answer as to why are they moving and can you, instead of moving them, create hard links in the new locations instead?

Here's an article that goes into a lot more detail about on the subject of snapshots; I think the answer to why you need to move files and if you have control of it, might help with this article???

share|improve this answer
    
Its mainly organizational, I have a couple of directory for incoming files, then when they are done being processed/whatever they are then stowed in an "done" directory. –  Pharaun Aug 23 '10 at 16:47
    
so on the system you are syncing FROM, I assume the files do not move then. Is it possible to keep both structures in sync at both ends then while using the hard link option? Part of what I do for a living is provide on-line backups for customers and we use this approach to keep a typical 7 days of retension in sync. –  Kilo Aug 26 '10 at 11:36
    
so even though the files at my customer's location do not move about, we keep their files in different directories going back day1, day2, day3,...,dayX. Hard linking and rsycing only the differences. Hope this helps. –  Kilo Aug 26 '10 at 11:38
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.