for f in `ssh $SSHCRED "~/file-list.sh"`
do
  rsync --progress --times --partial --append --rsh=ssh -r -h --remove-sent-files $SSHCRED:$f $OUTDIR
done

This is what I do today.

I'd like to instead do this by sending the output from ssh $SSHCRED "~/file-list.sh" directly to rsync. That should be faster, and I can easily abort just one rsync operation.

But I see at least one problem with this: The remote script generates a list of files with absolute paths. For rsync to work, I think I need to prepend the SSH credentials before every path, like this:

user@host:/path/to/file1
user@host:/path/to/file2
user@host:/path/to/file3

Is this possible?

link|improve this question

feedback

1 Answer

up vote 5 down vote accepted

Generate a list of files and hand them over to rsync with the --files-from=FILE option:

rsync --your-options --files-from=filelist.txt user@host:/basedir targetdir

This should be much faster than calling rsync for every single file.

link|improve this answer
Thanks! It almost works the way I want. The only problem is that it recreates the whole dir structure. So say I want to rsync every file in /home/backups to /var/backups on my local machine, it will create /var/backups/home/backups. Can I avoid this? – Znarkus Dec 2 '11 at 14:14
Yes. Make sure the list has only relative filenames (subdir1/filename1 instead of /home/backups/subdir1/filename1 and then use user@host:/home/backups as from-target for rsync. – SvenW Dec 2 '11 at 15:31
Oh okay. That makes it more complicated, if I want the remote script to handle what files should be copied from where. Even if it's just from one folder. Maybe I could extract the folder path from the first line, then remove that string from every line. – Znarkus Dec 3 '11 at 10:11
Or after the script has run, I could flatten the directory structure created by rsync, and leave a single folder with all files. – Znarkus Dec 3 '11 at 10:13
feedback

Your Answer

 
or
required, but never shown

Not the answer you're looking for? Browse other questions tagged or ask your own question.