17

ftp supports the put "|..." "remote-file.name" command to pipe data to an ftp connection. Is there something similar available for sftp?

In sftp i get the following error:

sftp 'jmw@backupsrv:/uploads'
sftp> put "| tar -cx /storage" "backup-2012-06-19--17-51.tgz"
stat | tar -cv /storage: No such file or directory

as above the sftp client doesn't obviously execute the command.

i want to use the pipe command to directly redirect the file stream to sftp. (because there is not enough space left to create a backup file on the same disk before uploading it to sftp server.)

4

4 Answers 4

9

As this is the first result you find by googling for this question and it hasn't been mentioned already I'll add the solution I found here as well:

you can use curls sftp implementation for this. As curl is probably already installed on a lot of systems this might be preferred to the solution using custom clients.

example usage:

pg_dump -d database | pigz -1 | curl -u username -T - sftp://sftpserver/folder/dbbackup.sql.gz

curl uses your .ssh/known_hosts file for key verification. This might fail in case your ssh client does use newer encryption standards not supported by the library used in curl

to fix this you can add the other key types to the known hosts file using the following command:

ssh-keyscan sftpserver >> ~/.ssh/known_hosts

or you can disable key verification using the -k flag (I wouldn't recommend that though)

4

I had a lot of fun figuring out a solution to this problem. It requires the tool nc (netcat) on both machines, and SSH (SFTP isn't needed).

In this example, I shall call the machine that has the data that needs to backed up linux-a, and the machine that needs to receive the backup linux-b.

On linux-a, have netcat listen on a port (I took 2000) and redirect it to a file. This will just sit there and wait until something comes through on that port.

[kenny@linux-b /var/backups]$ nc -l 2000 > backup.tgz

On linux-b, open up an ssh tunnel to linux-a, I used port 2000 again. This will redirect anything you throw at TCP port 2000 on localhost to TCP port 2000 on linux-a, where netcat is listening.

[kenny@linux-a /var/data]$ ssh -L 2000:localhost:2000 -CfN linux-b

Now create the tar archive, but send the output to stdout (using -) and pipe it to gzip for some compression. Now pipe that to another netcat that sends it to localhost on TCP on port 2000.

[kenny@linux-a /var/data]$ tar cf - important-data | gzip -fc | nc localhost 2000

We're done! On linux-b the netcat is no longer listening, and a new file is created. The best part is that the tar archive was never placed on the hard disk of linux-a.

[kenny@linux-b /var/backups]$ file backup.tgz 
backup.tgz: gzip compressed data, from Unix, last modified: Thu Jul  5 13:48:03 2012

I know it's not exactly what you asked for in the question, but if you have netcat available, its a viable solution to your type of problem.

Edit: I forgot about one thing: if you follow these instructions, you'll still have an SSH tunnel floating around on linux-a. Find out what the process ID is and kill it.

[kenny@linux-a /var/data]$ ps -ef | grep "ssh -L"
kenny     5741     1  0 13:40 ?        00:00:00 ssh -L 2000:localhost:2000 -CfN linux-b
kenny     5940  3360  0 14:13 pts/1    00:00:00 grep --color=auto ssh -L
[kenny@linux-a /var/data]$ kill 5741
3
  • 4
    i like it, but there are 2 problems: sftponly users usually cannot use port forwarding and sftponly users cannot access nc on the sftp server (remember they do not have ssh access)
    – JMW
    Jul 5, 2012 at 12:23
  • 1
    That's entirely valid criticism. Jul 5, 2012 at 12:27
  • Off-topic, but why is there a screen shot of the "The Prestige" embedded in this post?
    – Rilindo
    Jul 5, 2012 at 14:10
2

output-stream-generating-command | ssh user@remotehost 'input-stream-accepting-command' is an option, if your remote user has a valid shell.

8
  • 6
    This is NOT sftp!
    – jirib
    Jun 19, 2012 at 18:23
  • 3
    @JiriXichtkniha No, it isn't, however SFTP is almost always implemented as a subsystem of your SSH server, and this is the closest analog to the functionality I see being requested from the question (pipe data through SFTP to be handled by a program on the other end). Sometimes the answer is "You're using the wrong tool - do this instead." - this strikes me as one of those times.
    – voretaq7
    Jun 19, 2012 at 18:31
  • 2
    sftp is protol, and is different from just piping via ssh, that's it! Your suggestion would not work if remote part would be SFTP only.
    – jirib
    Jun 19, 2012 at 19:03
  • 3
    i'm not allowed to login as ssh user
    – JMW
    Jun 22, 2012 at 7:23
  • 1
    @JMW in that case as far as I know you're pretty well screwed here - I've never seen a version of sftp that supports piping to a command on the remote system (probably because it would defeat the Secure bit by letting someone in your situation who doesn't have a shell run programs anyway). I could be wrong though - I'm mostly familiar with the SFTP functionality in OpenSSH and the proprietary SSH server Sun used to ship...
    – voretaq7
    Jun 22, 2012 at 16:00
1

voretaq7 pointed out, that the sftp client doesn't support piped data transfer for users, that are allowed to use sftp only to connect to the server.

fortunately, there is libssh2, that supports sftp. so we just need 2 other clients using libssh2, that i've called:

  • sftp_stdin_upload (to upload to a sftp server)
  • sftp_stdout_download (to download from a sftp server)

the source code can be found under the following URL: http://www.qxs.ch/2012/07/05/sftp-upload-tool/


since i'm not that experienced in libssh2 programming, i'm happy for any feedback to the source code.

3
  • Doesn't work for me :-( /usr/bin/ld: libssh2-1.4.2/src/.libs/libssh2.a(knownhost.o): undefined reference to symbol 'EVP_sha1' /usr/lib/libcrypto.so.1.0.0: error adding symbols: DSO missing from command line collect2: error: ld returned 1 exit status
    – tobixen
    Jun 5, 2015 at 9:50
  • This code has just a POC status. Feel free to fix it on your own. :-)
    – JMW
    Jun 14, 2015 at 20:45
  • Uploading using libssh and libssh2 is quite slow in comparison to using sftp command-line client... Also, the link (to qxs.ch) returns a 404 today...
    – Mikhail T.
    Jul 28, 2022 at 21:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .