4

I'm trying to sync files from System A to System B. However, the files are re-organized in another directory structure, which makes usage of rsync difficult.

Is there any way to tell rsync to ignore directories and operate on file names only? The file names are unique - the directories aren't. The directory structure isn't fixed, so I can't simply replace them. I already thought about writing a script which strips the directory information, but I'm not sure if that brings up other problems.

In fact, yes, I wish to flatten the directory structure. Given the answers, rsync is probably not what I wish to use.

I'm working with videos, third parties create a directory structure (and they should be allowed to change the directory structure whenever appropriate). Those videos need to be syncronized to a master file system. File names are agreed not to be changed. So something like a diff between "find . | rip-out-path" on both systems and a diff might do the trick; but I was wondering if rsync had some magic flag to ignore directories at all when recursing - similar to the -p parameter in patch.

6
  • 1
    So, if I understand you, you've got lots of files in a consistent directory structure A and some identically named files at B; but in a somehow unpredictable arrangement each time you need to sync? I'm curious how this situation has come about? Also, do you need to keep the two disparate directory structures? (meaning you can't flatten the filesystem at both sides before you sync) Jul 27, 2011 at 18:27
  • Wait what? Do you want rsync to just not recurse? Or is something more complex going on? Can you provide example data?
    – MikeyB
    Jul 27, 2011 at 18:28
  • @SmallClanger yes, third parties create video projects on their disks and I need to sync them into a master file storage. This is done manually, but the amount of videos has grown so huge that this can't be done manually anymore. Teaching all 3rd parties to use a specific directory structure is more work than inventing some script. Jul 28, 2011 at 10:26
  • Do the files on System A change and require to be updated on System B or is it just a "copy once" of files on System A which are missing on System B? And a second point: are all the files on System B in the same directory, or do sub directories exist?
    – Marcel G
    Jul 28, 2011 at 10:59
  • @Felicitus: I'd advise trying to sort out the filesystem layout instead. If files are badly organised, then you need to address that, first. Keep it simple, rather than piling more complicated configuration on top of it. That said, if you're pulling from multiple client folders to a single, central location, can't you at least create a holding folder named after each machine or user, so there's no clashes, and then rsync as normal? Jul 28, 2011 at 11:46

10 Answers 10

2

Probably the simplest way to solve moving all files from on directory tree to a single directly would be using find with the -type and -exec options. The -type option limits the output to a specific type of directory entry (f for file, d for directory, etc.). The -exec option passes the name found (as {}) to a command line with options.

A couple examples follow:

find /directory/top/ -type f -exec rsync {} desthost:/destdir 

find /directory/top/ -type f -exec scp {} desthost:/destdir 
1

You're screwed, or or less. Whilst you can tell rsync to recurse and all sorts of other games, you can't tell it to go hunting around in a filesystem tree to find a file named the same at the other end.

I'd say what you're going to have to do is to have a little wrapper script at the far end that, given a bare file, returns the fully-qualified path to the file at that end, and then iterate through each file at the local end, calling this wrapper script to get the remote path and then executing rsync one... file... at... a... time...

That is, of course, assuming that all the files even already exist at the far end... where do they get put if they're not even there? Are they skipped?

I'd find whoever came up with this crackpot file storage scheme and break their fingers.

1
  • 1
    Different people use different directory structures. I'm handling video storage done by others, and need to sync new stuff into a master filesystem. Currently this is done by hand, and I wish to automate that. No need to break anyone's fingers. Jul 28, 2011 at 10:23
1

I'm still sticking with my "finger breaking" other answer in the general case, but I have a different solution for your specific situation, which is, as I understand it:

  • Other people have their own copy of stuff, in whatever crackpot hierarchy they choose; and
  • You need all their files, but organised into your own crackpot hierarchy

What I'm thinking is you run an rsync into remote-specific directories (like /storage/.remotes/client1/, /storage/.remotes/client2/, etc) for each of the remote filesystems you're syncing, and then have a script which normalises the filenames into your own hierarchy (assuming you can algorithmically describe your organisational scheme), and which you run over everything after the rsync has done it's thing to symlink into the client-specific remote storage locations. If you can't describe your desired hierarchy algorithmically, then I guess you'll have to do your symlinking by hand (or at least with some level of human input, even if there is tool support).

The only difficulty then is if the remote rearranges their stuff, but then you just detect the now-broken symlinks, find the new locations of the filenames (assuming the names haven't changed, just the locations).

1
SOURCE_DIR=/path/to/lots/of/dirs/and/files
LINK_PATH=/path/to/store/all/files/as/symlinks/in/single/directory
DEST_PATH=/path/to/place/all/files/in/single/directory/with/no/child/directories

find $SOURCE_DIR -type f -print0 | xargs -0 cp -s --target-directory=$LINK_PATH
rsync -Lts $LINK_PATH/* $USER@$DEST_IP:$DEST_PATH
1
  • This is almost what I was thinking: don't you need to create a link_path on both the local and the remote hosts before attempting to rsync?
    – sage
    Jun 10, 2014 at 23:55
1

You can copy files from various source folders into one target folder (flat) without transferring source subfolders using command:

find source_dir -name "*.pdf" >/tmp/xx.txt
rsync -t -v --no-relative --files-from=/tmp/xx.txt / desthost:/destdir
rm -f /tmp/xx.txt
0

what about the --fuzzy option in rsync? I dont know if it would work in your case but you could give it a try.

1
  • No, that's just for reducing the amount of data transferred; it does nothing about storing files in different places.
    – womble
    Jul 28, 2011 at 13:53
0

If all files are on the same filesystem, it might be easier to hardlink them all to one directory on the source side and then rsync that one directory across. Something like:

#!/bin/bash
set -e
mkdir flattened_dir
find sourcedir1 sourcedir2 sourcedir3 -type f -exec ln -t flattened_dir/ {} +
rsync -avP flattened_dir/ remote:destination/
rm -r flattened_dir

P.S. If find does not support +, you can use \;

0

Similarly to this, I wanted to pull files out of directories and put them in a single flat directory using just their filename. The solution is:

 find /directory/top/ -type f -exec rsync -av `basename {}` desthost:/destdir 

You can also use some of the other flags in find to limit what files you want... like maybe you only want the JPG files:

 find /directory/top/ -type f -name "*.JPG" -exec rsync -av `basename {}` desthost:/destdir 
0

You can make use of Bash globopt (**) to match each file recursively, as described in this post.

Since you're only invoking rsync only once, it should be much faster than other approaches where you invoke a command for each file (like find ... -exec).

-1

You could do one dir at at time if that gives you a good batch size of files in one process call to rsync. So something like:

find . -type d | while read dir; do rsync -a $dir/* user@host:flatdir; done

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .