82

Rsync over ssh, works great every time.

However, trying to rsync to a host which allows only sftp logins, but not ssh logins, provides the following error:

rsync -av /source ssh user@remotehost:/target/

protocol version mismatch -- is your shell clean? (see the rsync man page for an explanation) rsync error: protocol incompatibility (code 2) at compat.c(171) [sender=3.0.6]

Here's the relevant section from the rsync man page:

This message is usually caused by your startup scripts or remote shell facility producing unwanted garbage on the stream that rsync is using for its transport. The way to diagnose this problem is to run your remote shell like this:

          ssh remotehost /bin/true > out.dat

then look at out.dat. If everything is working correctly then out.dat should be a zero length file. If you are getting the above error from rsync then you will probably find that out.dat contains some text or data. Look at the contents and try to work out what is producing it. The most com‐ mon cause is incorrectly configured shell startup scripts (such as .cshrc or .profile) that contain output statements for non-interactive logins.

Trying this on my system produced the following in out.dat:

ssh-dummy-shell: Command not allowed.

As I thought, the host is not allowing ssh logins.

The following link shows that it is possible to accomplish this task using fuse with sshfs - however it is extremely slow, and not fit for production use.

Is there any chance of getting rsync sftp to work?

Improve this question
5
  • 3
    lftp will do the trick. I will sync local to remote with the -R switch
    – user175622
    May 28, 2013 at 17:09
  • Do you know how the commands are restricted on the server? If it is by setting ForceCommand in sshd_config, then see if you can modify the file that points to. Or if it's set by command in authorized_keys, then maybe you can access authorized_keys over sshfs and change it?
    – ptman
    Apr 2, 2014 at 10:56
  • You could try csync: csync /home/csync sftp://[email protected]:2222/home/csync See this answer.
    – nachtigall
    Sep 6, 2014 at 4:45
  • 8
    Eventually, I did it with lftp: lftp -e "mirror -eRv /local/dir/ /remote/dir; quit;" sftp://user@server:port
    – nachtigall
    Sep 6, 2014 at 5:51
  • @nachtigall do you know how to use lftp with multiple input files / a list of files and dirs?
    – bortran
    Apr 4, 2016 at 14:10

8 Answers 8

Reset to default
45

Unfortunately not directly. rsync requires a clean link with a shell that will allow it to start the remote copy of rsync, when run this way.

If you have some way of running long-lived listening processes on the host you could try starting rsync manually listening for connections on a non-privileged port, but most techniques for doing that would require proper shell access via SSH too, and it relies on the hosts firewall arrangements letting connections in on the port you chose (and the host having rsync installed in the first place). Running rsync as a publicly addressable service (rather than indirectly via SSH or similar) is not generally recommended for non-public data though.

If you host allows scripting in PHP or similar and does not have it locked down so extra processes can not be execed by user scripts, then you could try starting rsync in listening mode that way. If your end is connectible (you are running SSH accessible to the outside world) you could try this in reverse - have a script run rsync on the server but instead of listening for incoming connections have it contact your local service and sync that way. This still relies on rsync actually being installed on the host which is not a given, or that you can upload a working copy, but does not have the security implications of running an rsync daemon in a publicly addressable fashion and talking to it over an unencrypted channel.

Messing around as described above may be against the hosts policies though, even if it works at all, and could get you kicked off. You are better off asking if a full shell can be enabled for that account and either abandoning rsync for that host or abandoning that host and moving elsewhere if they will not do that.

Improve this answer
1
  • 2
    Note that it doesn't have to be a full shell; it just has to allow the rsync command with the proper arguments to run. scponly is useful for this.
    – sciurus
    Sep 20, 2011 at 14:15
32

An alternative to using rsync is to instead use lftp (which can connect to sftp) and use the mirror command. For example one can do

lftp
~> open -u user,password sftp://host.com
~> mirror remotedir outdir
~> quit
Improve this answer
3
  • 3
    While this post does not answer the question it provides a helpful alternative
    – Dima Chubarov
    Apr 4, 2019 at 10:52
  • 1
    rclone is another alternative but with a more automation-friendly CLI.
    – ominug
    Apr 15, 2021 at 18:02
  • 1
    I tested your advice, but rclone is very slow with sftp. lftp performance are better.
    – Jérôme B
    May 20 at 8:56
25

Theoretically, yes. You can mount the remote filesystem on your local machine using FUSE. Then you can run a local copy of rsync between the mounted directory and the local directory. I've not personally tried this, but it should work in theory. It would likely be a lot less efficient that performing the rsync over SSH because it would need to transfer at least part of each file to perform the comparison.

Improve this answer
3
  • 14
    That should work when timestamp+size comparison is enough to decide what needs to be send. As soon a a checksum is needed the entire remote file will be read over the line as well as the updates sent, so make sure you have rsync's options set such that it does all-or-nothing for each file (not using --ignore-times or --checksum and having --whole-file specified may be enough). Some options like --link-dest are highly unlikely to work properly (or at all) this way.
    – David Spillett
    Apr 26, 2010 at 1:11
  • Thanks for adding some details David. That's the kind of limitations in efficiency I was thinking of, just didn't articulate them quite as well :)
    – Kamil Kisiel
    Apr 26, 2010 at 17:42
  • It's a workaround with limitations, yes, but it is good enough if there is no way to invoke "proper" rsync
    – Valery Lourie
    Jul 13, 2014 at 9:15
14

Instead of Rsync, rclone supports syncing from/to and mounting SFTP network shares.

  1. Install rclone
  2. rclone config and follow the steps to add a new remote endpoint configuration. Input sftp at the protocol backend selection.
  3. rclone sync local_path remote:remote_path (docs). This can cause data loss.

IMO, rclone provides a more automation-friendly CLI than LFTP.

I know this does not answer the original question but probably the needs of some people which come here.

Improve this answer
4
  • Good outside-box thinking -- can confirm that rclone works where rsync fails to do the job!
    – ctietze
    May 3, 2022 at 17:02
  • cool written in go (modern language that's easier to maintain), 35k github stars, and actively developed. (csync looks like abandonware, and lftp looks like dev has stopped due to global events.) Thanks for the intro to what looks like a promising tool.
    – neoakris
    Oct 13, 2022 at 20:55
  • 3
    THIS COMMAND CAN DELETE FILES ON THE DESTINATION. You may wish to use rclone copy instead, which more closely matches rsync's default behaviour.
    – mxbi
    Jan 1 at 14:06
  • 1
    Also rclone is very slow compared to lftp
    – Jérôme B
    May 20 at 9:07
6

a bit late, but here is how I do it, using sshfs

  source /scratch/slimdata/password.sh
  mkdir tmp_mnt
  echo $PASSWORD | sshfs user@host:dir tmp_mnt -o password_stdin
  rsync -rutL --delete tmp_mnt/ to_sync/
  fusermount -u tmp_mnt
  rmdir tmp_mnt
Improve this answer
2
  • 4
    As @David Spillett says in his comment on an earlier answer (serverfault.com/questions/135618/…), this will work but is a bit of an inefficient use of the rsync protocol.
    – Evan Anderson
    Sep 20, 2011 at 12:41
  • 3
    jonathan, I think your solution is wrong. If you mount a remote drive and execute rsync like you would sync two local folders, there is always transfered everything on syncing since the rsync process on your local machine calculates the hash values of the remote files. This means the whole file is transfered and the benefits of rsync are lost. Edit: ah this answer has been given several times... sry for replication :D
    – Thekwasti
    Mar 22, 2012 at 10:20
5

Option combining best of all

It appears to have the following advantages that other answers here don't have:

  • fully benefits from SSH (secure)
  • fully benefits from rsync (bandwidth-efficient protocol, all rsync options like bandwidth limitation)
  • actually efficient (unlike sshfs which save the days sometimes but is still slow in practice)
  • doesn't need arbitrary shell commands server-side
  • doesn't need server to allow ssh tunnels
  • doesn't need server to run a rsync daemon

I have not yet tested it, but I've successfully used all those features except rrsync.

How to do it

  • create a key locally with ssh-keygen (openSSH feature)
  • allow login only with that specific key, with an entry in server's ~/.ssh/authorized_keys (openSSH feature)
  • associate this key to a specific command, with ~/.ssh/authorized_keys (openSSH feature) using as command the script rrsync distributed with rsync, something like command="$HOME/bin/rrsync -ro ~/backups/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-

Then you can rsync from client normally.

If you need more details

Restricting SSH Access to rsync | Guy Rutenberg

One step beyond link above

The command at the end of that page can be made shorter. In ~/.ssh/config create a stanza like this:

Host remote # can be host or ip or custom-label User user # login on remote host HostName optional-dns-resolvable-host-or-ip # if label used above IdentityFile ~/.ssh/id_remote_backup

then your rsync command from client

rsync -e "ssh -i $HOME/.ssh/id_remote_backup" -av user@remote: etc2/

becomes

rsync -av user@remote: etc2/
Improve this answer
2
  • 7
    Unfortunately this doesn't have anything to do with using rsync with an SFTP server though, this is about configuring an OpenSSH server to allow rsync to be run, sidestepping the SFTP aspect. If you are presented with someone else's SFTP server that you don't have any control over then this answer can't solve the problem.
    – Malvineous
    Sep 7, 2018 at 6:14
  • You're right. I might have actually written this to answer another question and posted here by mistake. I'm considering removing my answer here.
    – Stéphane Gourichon
    Sep 7, 2018 at 15:31
3

No. rsync works by running rsync on the other side and communicating with it, which means that some form of shell access is required.

Improve this answer
2

An alternative would be to start rsync as daemon and connect to it through a SSH tunnel.

Improve this answer
1
  • This answer is too short to be completely helpful, but if the server side cannot allow shell access but can allow (1) a rsync server listening locally (not available from the outside, for security), and (2) ssh tunnel, then in principle one could log in to server via a rule in authorized_keys that allows a specific tunnel and runs e.g. sleep 86400. Then the client side could then rsync via the tunnel.
    – Stéphane Gourichon
    Jan 8, 2018 at 9:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .