Moving files between hosts using intermediate storage area [closed]

Question

Is there any good existing solution for moving files from host A to host C, using host B a an passive intermediate storage area.

A -> B -> C

Limitations

• Host A and host C are on separate networks and can not access each other. No tunnels can be opened between A and C.
• Both host A and host C can run and schedule periodic processes.
• Host B only provides a storage area and can be accessed from both host A and host C through SSH.
• B will be "passive" and will only provide the storage area.
• Thus A will act as writer (of files on B), and C will act as a reader (of files on B).
• Files that are successfully copied from host A to host C (via B) can then be deleted on host A.
• No files on host A should be deleted until there is a guarantee (a receipt containing a file checksum?) it has successfully landed on host C.
• Corner cases includes management of network downtime, disk full, interrupted processes etc

The aim is to, at regular intervals, move files from A to C, using B as intermediate storage area.

We can assume that all host are running Unix/Linux. Tools preferred: ssh/rsync/bash + other "standard" tools.

I was thinking of creating a solution by using some bash scripts around rsync in two steps and using receipts containing checksums to detect when a file is successfully moved all the way.

Are there any existing solutions?

Answer 1

I don't see the need to store anything on B, you can just use it to proxy connections between A and C using any proxy software, like ssh port forwarding or socks. Maybe you could even route/nat through B?

Or, since you specifically asked, to do it your way you could use NFS/Samba to copy the files and verify completion using various methods. The one that comes to mind would be to have a file created on the share to pass "copy complete" or "ok to delete" type messages between both systems. Or like you said, something with MD5, etc... However you do it, I'm pretty sure it will be a custom job. But not a difficult one. I doubt there are any existing ways of dealing with your problem in that manner.

Answer 2

Sven's comment to your question is definitely the right way to do it.

You can do it like this:

1. From A SSH to B, creating a tunnel on port 5000

   ssh someuser@B -L 5000:127.0.0.1:5000
   

2. From that shell on B, SSH to C creating a second tunnel terminating at C's ssh server.

   ssh someuser@C -NL 5000:127.0.0.1:22
   

3. From a second terminal on A, rsync your files directly to C

   rsync -avz -e "ssh -p 5000" --progress --remove-source-files /source/directory [email protected]:/destination/directory
   

It might even be possible to setup the tunnels in step 1 and 2 with one command, but in the quick test I did, I wasn't able to. It might require setting up ssh keys on B and C.

If you're going to automate the copy process, then you might need to look at something like autossh to setup and teardown the tunnels for you via services. Or you could do it via cron..