15
votes

Basically I am searching to see if there exists a tool or script that can detect moved or renamed files so that I can get a list of renamed/moved files and apply the same operation on the other end of the network to conserve on bandwidth.

Basically disk storage is cheap but bandwidth isn't, and the problem is that the files often will be reorganized or moved around into a better directory structure thus when you use rsync to do the backup, rsync won't notice that its a renamed or moved file and re-transmission it over the network all over again despite having the same file on the other end.

So I am wondering if there exists a script or tool that can record where all the files are and their names, then just prior to a backup, it would rescan and detect moved or renamed files, then I can take that list and re-apply the move/rename operation on the other side.

Here's a list of the "general" features of the files:

  1. Large unchanging files
  2. They can be renamed or moved around

[Edit:] These all are good answers, and what I end up doing in the end was looking at all of the answers and will be writing some code to deal with this. Basically what I am thinking/working on now is:

  1. Using something like AIDE for the "initial" scan and enable me to keep checksums on the files because they are supposed to never change, so it would aid on detecting corruption.
  2. Creating an inotify daemon that would monitor these files/directory and recording any changes relating to renames & moving the files around to a log file.
  3. There are some edge cases where inotify might fail to record that something happened to the file system, thus there is a final step of using find to search the file system for files that has a change time latter than the last backup.

This has several benefits:

  1. Checksums/etc from AIDE to be able to check/make sure that some media did not get corrupt
  2. Inotify keeps resource usage low and no need to re-scan the filesystem over and over
  3. No need to patch rsync; If I have to patch things I can, but I would prefer to avoid patching things to keep the burden lower, (IE don't need to re-patch everytime there is an update).
  4. I've used Unison before and its really nice, however I could've sworn that Unison does keep copies around on the filesystem and that its "archive" files can grow to be rather large?
0

9 Answers 9

7
votes

Unison http://www.cis.upenn.edu/~bcpierce/unison/ claims to be able to detect moves and renames.

There are a couple patches to rsync to add move/rename detection:

http://gitweb.samba.org/?p=rsync-patches.git;a=blob;f=detect-renamed-lax.diff;h=1ff593c8f97a97e8970d43ff5a62dfad5abddd75;hb=master

http://gitweb.samba.org/?p=rsync-patches.git;a=blob;f=detect-renamed.diff;h=c3e6e846eab437e56e25e2c334e292996ee84345;hb=master

Bugzilla entry tracking this issue: https://bugzilla.samba.org/show_bug.cgi?id=2294

1
  • 6
    Why aren't these patches integrated? They just add flags, they're not intrusive. Another interesting patch is rsyncsums, which can keep checksums around between rsync runs.
    – Tobu
    Nov 12, 2010 at 21:44
5
votes

This is a bit of an odd solution, but... git detects moves and renames based on file content, so if you were to keep the directories in question under version control then git would be able to detect moves and such and avoid transferring the content (since it's already on both sides of the wire) while still moving things around in the tree.

Just a thought.

3
  • 2
    Yes I considered this, if the files were small and text based, this would probably work good, but they are binary and the total size is approaching a Terabyte.
    – Pharaun
    Aug 18, 2010 at 16:21
  • @Pharaun You'd need the git index without the blob storage. Maybe rip this code out of git and add it to libgit2.
    – Tobu
    Nov 12, 2010 at 21:58
  • The relevant code starts with refresh_index in read-cache.c .
    – Tobu
    Nov 12, 2010 at 22:03
5
votes

interesting suggestions here. Also thought of using filesystem capabilities ie ZFS. Found it strange that there is no tool which does that simple thing. Unison option does not work in most cases as people report, not for me either.

I want the feature to keep backup of my movie collection on second hard disk in sync when rearraring folders.

Now i found this simple C script http://sourceforge.net/projects/movesync/

Seems to work fine. Run it and then sync normally with ie unison.

4
votes

You might be able to use a host based IDS such as AIDE and write a wrapper script using its output. You would likely have to write more complex logic considering the checksums.

Otherwise, a network based filesystem might make sense, as the changes would be reflected at all locations. Nevertheless, I suspect you are transferring over the Internet, which will limit options here.

1
  • That was what I was thinking of doing, taking one of those and extend them. Also yes I am transferring it over the internet and the bandwidth is pretty limited.
    – Pharaun
    Aug 18, 2010 at 15:41
3
votes

You might try unison ; especially the

-xferbycopying optimize transfers using local copies (default true)

option mentioned in the docs as

When this preference is set, Unison will try to avoid transferring file contents across the network by recognizing when a file with the required contents already exists in the target replica. This usually allows file moves to be propagated very quickly. The default value is true.

looks like it might do what you want.

1
  • Actually in hindsight, I might have been too hasty on the unison comment. Does unison support replacing a hardlink with the actual file content if it changes? If so then I might be able to do some magic with rsnapshot + unison which would meet my requirements without having to write up a ton of new code/log/etc to deal with this.
    – Pharaun
    Aug 23, 2010 at 15:42
3
votes

Syrep does what you need. It keeps message digests on a file tree up to date; keeping the digests around makes it more efficient than rsync. It was designed for sneakernet so you may want to add a wrapper that does update/makepatch/merge at once.

2
votes

I'm not sure if there's an existing tool that does this for you, but you could write a simple script that just runs a find on the base directory where mtime is newer than the last backup. This will get you a list of all files that have been modified. If a file was simply moved, it will not appear in the list. Unfortunately, this list will include the directories that the files moved into, since the directory gets updated when a file is added/removed.

With that list of files, you can use rsync to only sync those files. rsync has an option to read in a file list. Here's a test showing this example:

$ cd tmp
$ echo test > test
$ ls -la
total 16
drwxr-xr-x 2 root root 4096 Aug 18 11:34 .
drwxr-x--- 5 root root 4096 Aug 18 11:34 ..
-rw-r--r-- 1 root root    5 Aug 18 11:34 test
$ mkdir tmp2
$ find . -mmin 1
$ date
Wed Aug 18 11:35:10 EDT 2010
$ find . -mmin 1
$ find . -mmin 2
.
./test
./tmp2
$ mv test tmp2
$ find . -mmin 1
.
./tmp2

Please note that I waited approximately 1 minute between running each find command. From this, it shows that when initially creating the file, it gets listed by find. If I move the file into another directory and re-run the find command, it only displays the directory I moved the file into, and not the file itself. You can use a combination of find and rsync commands to only list the files you want, it can probably achieve your goal.

I hope this helps.

2
votes

Given your workflow, I wonder if working at the file level (like what others have proposed so far) is the best solution. You could work...

At the filesystem level

The idea is to have the filesystem keep track of operations between backups. Instead of making a backup of the filesystem, back up the filesystem journal (and optionally replay the changes on the backup machine, if you want a ready-to-use backup). A filesystem journal naturally expresses moves and deletions in a few bytes.

Fuse makes it relatively easy to design a filesystem with specific requirements that sits on top of a “real filesystem”. I've never used it, but LoggedFS looks promising.

With this solution, it would be worthwhile to have some form of journal compression. For example, if a file has been overwritten 10 times, only keep its last update in the journal. Another worthwhile optimization would be to recognize copy operations, and even better, edits (i.e., creating a file that is mostly but not completely identical to another file). I don't know if anybody has implemented this. For your workflow, I don't think it would matter much anyway.

At the volume level

The idea is to have the volume manager keep track of operations between backups. Instead of making a backup of the filesystem, take a snapshot with the volume manager and back up the snapshot expressed as a diff from the previous snapshot.

This should work well if all you do is create files, rename them and remove them. It would be a lot harder to detect things like copies and edits, or to optimize away the creation of a file followed by its deletion.

2
  • I've actually been working a bit on a file "system" logger via inotify to keep track of changes, but if the changes comes in faster than the speed that the daemon can record it, it will lose information, hence need to build a backup/scan to get the initial state and in case of inotify losing information. It does look like the idea of having something that sits between the filesystem & rest of the system might also be a good idea then like you said, that changes could be replayed on the backup machine.
    – Pharaun
    Aug 24, 2010 at 13:45
  • But that loggedFS does look like an interesting project, only concern is they stopped dev in 2008/09. Going to have to play with it and see if it will do the trick.
    – Pharaun
    Aug 24, 2010 at 13:48
0
votes

Unison is good for this, but still needs to copy files locally and it cannot detect a move/rename if also the file content changed even a little.

I made a simple Python script to detect renamed/moved files and directories using inode numbers (*nix only) and replay these changes on the synchronized machine. You can use it by itself or as a "renaming preprocessor" for Unison or rsync. It can be found here

Not the answer you're looking for? Browse other questions tagged .