2

I am attempting to pull everything from a remote directory which is more then ten minutes old, and then remove the source files.

My remote find seems to be working (tested independently on the remote box), but when rsync goes to use it, my user's home path is being prefixed to the find output, like so:

rsync -avz --dry-run --remove-source-files --files-from=<(ssh spot@host "find /mnt/volume_2/partners/test/uploads/ -amin +10 -type f,d -print0") spot@host:. .

receiving incremental file list
rsync: [sender] link_stat "/home/spot/mnt/volume_2/partners/test/uploads" failed: No such file or directory (2)
rsync: [sender] link_stat "/home/spot/mnt/volume_2/partners/test/uploads/file-1054" failed: No such file or directory (2)
rsync: [sender] link_stat "/home/spot/mnt/volume_2/partners/test/uploads/file-1105" failed: No such file or directory (2)
rsync: [sender] link_stat "/home/spot/mnt/volume_2/partners/test/uploads/file-1105/subfile-1105" failed: No such file or directory (2)
rsync: [sender] link_stat "/home/spot/mnt/volume_2/partners/test/uploads/file-1101" failed: No such file or directory (2)
rsync: [sender] link_stat "/home/spot/mnt/volume_2/partners/test/uploads/file-1056" failed: No such file or directory (2)
rsync: [sender] link_stat "/home/spot/mnt/volume_2/partners/test/uploads/file-1058" failed: No such file or directory (2)

I've spent more time than I'd like to admit attempting to resolve this with no success. I am hoping someone here is nice enough to point me in the right direction.

Thank you!

1
  • Search for similar questions on Unix & Linux for more examples. – anx Nov 6 '20 at 21:21
5

rsync is looking at the wrong place, because those paths are relative (from the rsync man page, emphasis mine):

--files-from=FILE

[..]

The filenames that are read from the FILE are all relative to the source dir

Either send absolute paths to the files-from option, but do not additionally define a source directory other than the root (/). This should work - I replaced one character after the host name:

rsync -avz --dry-run --remove-source-files --from0 --files-from=<(ssh spot@host "find /mnt/volume_2/partners/test/uploads/ -amin +10 -type f,d -print0") spot@host:/ .

Or - and especially useful when you actually want to deal with relative paths in the destination - tell find to emit relative paths and actually use them:

ssh spot@host "cd /mnt/volume_2/partners/test/uploads/; find -amin +10 -type f,d -print0" | rsync -avz --dry-run --remove-source-files --from0 --files-from=- spot@host:/mnt/volume_sfo2_01/partners/test/uploads/ .

Note I switched the <() redirection to a normal | pipe for better interoperability and less surprises while escaping the command for inclusion in scripts. I also added --from0 on top of your comment because the input is now null-delimited so it makes sense to tell rsync it is so.

5
  • Thanks for your response. I did try this before and the issue I then have is that it creates the entire directory structure at the destination. It creates a mnt dir and then a volume_2 inside it, and so on. Is there a way to take this solution and have it create the file tree relative to the source path? – Spot Nov 6 '20 at 22:28
  • 1
    I think I figured this out. This seems to work for me: rsync -avz --files-from=<(ssh spot@host "cd /mnt/volume_2/partners/test/uploads/; find -amin +1 -type f,d -print0") spot@host:/mnt/volume_sfo2_01/partners/test/uploads/ . – Spot Nov 6 '20 at 22:40
  • 1
    You changed the -amin +10 into -amin +1, if that's a typo you probably want to fix it before removing --dry-run – anx Nov 6 '20 at 23:08
  • That was intentional for testing, but thanks for the heads up just in case! :) – Spot Nov 9 '20 at 16:05
  • 1
    Thanks for the update, that helps understanding the structure more. – Spot Nov 9 '20 at 16:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.