Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm trying to sync files from System A to System B. However, the files are re-organized in another directory structure, which makes usage of rsync difficult.

Is there any way to tell rsync to ignore directories and operate on file names only? The file names are unique - the directories aren't. The directory structure isn't fixed, so I can't simply replace them. I already thought about writing a script which strips the directory information, but I'm not sure if that brings up other problems.

In fact, yes, I wish to flatten the directory structure. Given the answers, rsync is probably not what I wish to use.

I'm working with videos, third parties create a directory structure (and they should be allowed to change the directory structure whenever appropriate). Those videos need to be syncronized to a master file system. File names are agreed not to be changed. So something like a diff between "find . | rip-out-path" on both systems and a diff might do the trick; but I was wondering if rsync had some magic flag to ignore directories at all when recursing - similar to the -p parameter in patch.

share|improve this question
1  
So, if I understand you, you've got lots of files in a consistent directory structure A and some identically named files at B; but in a somehow unpredictable arrangement each time you need to sync? I'm curious how this situation has come about? Also, do you need to keep the two disparate directory structures? (meaning you can't flatten the filesystem at both sides before you sync) –  SmallClanger Jul 27 '11 at 18:27
    
Wait what? Do you want rsync to just not recurse? Or is something more complex going on? Can you provide example data? –  MikeyB Jul 27 '11 at 18:28
    
@SmallClanger yes, third parties create video projects on their disks and I need to sync them into a master file storage. This is done manually, but the amount of videos has grown so huge that this can't be done manually anymore. Teaching all 3rd parties to use a specific directory structure is more work than inventing some script. –  Felicitus Jul 28 '11 at 10:26
    
Do the files on System A change and require to be updated on System B or is it just a "copy once" of files on System A which are missing on System B? And a second point: are all the files on System B in the same directory, or do sub directories exist? –  Marcel G Jul 28 '11 at 10:59
    
@Felicitus: I'd advise trying to sort out the filesystem layout instead. If files are badly organised, then you need to address that, first. Keep it simple, rather than piling more complicated configuration on top of it. That said, if you're pulling from multiple client folders to a single, central location, can't you at least create a holding folder named after each machine or user, so there's no clashes, and then rsync as normal? –  SmallClanger Jul 28 '11 at 11:46

8 Answers 8

SOURCE_DIR=/path/to/lots/of/dirs/and/files
LINK_PATH=/path/to/store/all/files/as/symlinks/in/single/directory
DEST_PATH=/path/to/place/all/files/in/single/directory/with/no/child/directories

find $SOURCE_DIR -type f -print0 | xargs -0 cp -s --target-directory=$LINK_PATH
rsync -Lts $LINK_PATH/* $USER@$DEST_IP:$DEST_PATH
share|improve this answer
    
This is almost what I was thinking: don't you need to create a link_path on both the local and the remote hosts before attempting to rsync? –  sage Jun 10 at 23:55

Similarly to this, I wanted to pull files out of directories and put them in a single flat directory using just their filename. The solution is:

 find /directory/top/ -type f -exec rsync -av `basename {}` desthost:/destdir 

You can also use some of the other flags in find to limit what files you want... like maybe you only want the JPG files:

 find /directory/top/ -type f -name "*.JPG" -exec rsync -av `basename {}` desthost:/destdir 
share|improve this answer

If all files are on the same filesystem, it might be easier to hardlink them all to one directory on the source side and then rsync that one directory across. Something like:

#!/bin/bash
set -e
mkdir flattened_dir
find sourcedir1 sourcedir2 sourcedir3 -type f -exec ln -t flattened_dir/ {} +
rsync -avP flattened_dir/ remote:destination/
rm -r flattened_dir

P.S. If find does not support +, you can use \;

share|improve this answer

You could do one dir at at time if that gives you a good batch size of files in one process call to rsync. So something like:

find . -type d | while read dir; do rsync -a $dir/* user@host:flatdir; done
share|improve this answer

Probably the simplest way to solve moving all files from on directory tree to a single directly would be using find with the -type and -exec options. The -type option limits the output to a specific type of directory entry (f for file, d for directory, etc.). The -exec option passes the name found (as {}) to a command line with options.

A couple examples follow:

find /directory/top/ -type f -exec rsync {} desthost:/destdir 

find /directory/top/ -type f -exec scp {} desthost:/destdir 
share|improve this answer

I'm still sticking with my "finger breaking" other answer in the general case, but I have a different solution for your specific situation, which is, as I understand it:

  • Other people have their own copy of stuff, in whatever crackpot hierarchy they choose; and
  • You need all their files, but organised into your own crackpot hierarchy

What I'm thinking is you run an rsync into remote-specific directories (like /storage/.remotes/client1/, /storage/.remotes/client2/, etc) for each of the remote filesystems you're syncing, and then have a script which normalises the filenames into your own hierarchy (assuming you can algorithmically describe your organisational scheme), and which you run over everything after the rsync has done it's thing to symlink into the client-specific remote storage locations. If you can't describe your desired hierarchy algorithmically, then I guess you'll have to do your symlinking by hand (or at least with some level of human input, even if there is tool support).

The only difficulty then is if the remote rearranges their stuff, but then you just detect the now-broken symlinks, find the new locations of the filenames (assuming the names haven't changed, just the locations).

share|improve this answer

what about the --fuzzy option in rsync? I dont know if it would work in your case but you could give it a try.

share|improve this answer
    
No, that's just for reducing the amount of data transferred; it does nothing about storing files in different places. –  womble Jul 28 '11 at 13:53

You're screwed, or or less. Whilst you can tell rsync to recurse and all sorts of other games, you can't tell it to go hunting around in a filesystem tree to find a file named the same at the other end.

I'd say what you're going to have to do is to have a little wrapper script at the far end that, given a bare file, returns the fully-qualified path to the file at that end, and then iterate through each file at the local end, calling this wrapper script to get the remote path and then executing rsync one... file... at... a... time...

That is, of course, assuming that all the files even already exist at the far end... where do they get put if they're not even there? Are they skipped?

I'd find whoever came up with this crackpot file storage scheme and break their fingers.

share|improve this answer
    
Different people use different directory structures. I'm handling video storage done by others, and need to sync new stuff into a master filesystem. Currently this is done by hand, and I wish to automate that. No need to break anyone's fingers. –  Felicitus Jul 28 '11 at 10:23

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.