Variable directory names over SCP

Question

We have a backup routine that previously ran from one disk to another on the same server, but have recently moved the source data to a remote server and are trying to replicate the job via scp.

We need to run the script on the target server, and we've set up key-based scp (no username/password required) between the two servers. Using scp to copy specific files and directories works perfectly:

scp -r -p -B bus[email protected]:/mnt/disk1/bsource/filename.txt /mnt/disk2/btarget/

However, our previous routine iterates through directories on the source disk to determine which files to copy, then runs them individually through gpg encryption. Is there any way to do this only by using scp?

Again, this script needs to run from the target server, and the user the job runs under only has scp (no ssh) access to the target system.

The old job would look something like this:

#Change to source dir
cd /mnt/disk1

#Create variable to store 
# directories named by date YYYYMMDD
j="20000101/"

#Iterate though directories in the current dir
# to get the most recent folder name
for i in $(ls -d */);
do
 if [ "$j" \< "$i" ]; then
  j=${i%/*}
 fi
done

#Encrypt individual files from $j to target directory
cd ./${j%%}/bsource/
for k in $(ls -p | grep -v /$);
do
 sudo /usr/bin/gpg -e -r "Backup Key" --batch --no-tty -o "/mnt/disk2/btarget/$k.gpg" "$/mnt/disk1/$j/bsource/$k"
done

Can anyone suggest how to do this via scp from the target system? Thanks in advance.

EDIT: With all the bash one-liners I see flying around here, I was half-expecting within a few seconds to see some epic thing pop up with cat, grep, and maybe a '>>' or two between a few pipes. Ah, well. ;-)

Answer 1

Run a similar version of the earlier script on the source server using cron, where it archives and encrypts all the files to a directory. At an appropriate interval following that, have a cron on the target server rsync over SSH that location. You could scp but rsync has many advantages.

Otherwise, I'd scrap your requirements and run an on-demand hybrid solution from the target using SSH pipes.

Answer 2

I don't know, if sshfs could be a solution for you. This way you wouldn't have to allow unrestricted ssh, and you wouldn't even have to change your bash script. (I guess the performance would be somewhat worse though, and maybe you have additional requirements that won't allow this approach.)

Answer 3

UPDATE: After some experimentation, I've worked out how to pipe scp to gpg from the target server via stdout:

scp -r -p -B [email protected]:/mnt/disk1/bsource/filename /dev/stdout | gpg -e -r "Backup Key" --batch --no-tty -o "filename.gpg" 

I think for the other half of this I'll run the sorting job on the source server and output the resulting directory name to a text file in a static location, which will be scp'd first from the target system, then cat'd into the variable for the source location.