187

Any unix:

I have the following cmd line which works fine.

rsync -avr -e ssh /home/dir user@example.com:/home/

But I need to set it up now to rsync to a remote server that only has an FTP server on it. How do I go about that?

I looked at the rsync help but quickly got lost (I don't do this stuff very often).

  • 21
    This is akin to asking how to use HTTP over IRC, or how to use FTP over SMTP... – Juliano Nov 11 '09 at 12:26
  • 2
    Maybe you could explain some of your reasoning for wanting to do this, then we can help you come up with a viable solution. – James Jul 5 '11 at 21:06
  • 1
    essentially i was trying to automate a backup from a shared hosting server, and didn't understand how rsync worked. i have moved to a vps now, and don't have the limitation of ftp anymore – bumperbox Jul 5 '11 at 21:58
  • Duplicity is your god: scie.nti.st/2013/4/13/… – user956584 Feb 27 '15 at 15:02
  • If deployment is, what's on your mind, and you don't mind NodeJS, check out dploy, lazy and works well with git – Frank Nocke Dec 12 '16 at 14:42

18 Answers 18

78

You don't. rsync can't do that for you, it is a protocol of its own and doesn't work over FTP.

You might, however, want to try csync. IIRC it provides rsync-like behaviour over HTTP. I can't comment on whether it works over FTP, you'll have to try it.

162

rsync isn't going to work for you for the reasons others have mentioned. However, lftp and ncftp both have "mirror" modes that will probably meet your needs.

I use this to push stuff from my local directory to a ftp or sftp web host:

lftp -c "set ftp:list-options -a;
open ftp://user:password@your.ftp.com; 
lcd ./web;
cd /web/public_html;
mirror --reverse --delete --use-cache --verbose --allow-chown  
--allow-suid --no-umask --parallel=2 --exclude-glob .svn"
  • 2
    lftp is very nice. another one i just found out about is wput which has built-in bandwidth limiting. – karmakaze Jul 17 '13 at 6:46
122

As written by easel, lftp is a good tool.

I suggest you to parametrize the script, and make use of the

exclude-glob

options, that excludes filenames using the glob feature (*,? ..) of your shell:

#!/bin/bash    
HOST="your.ftp.host.dom"
USER="username"
PASS="password"
FTPURL="ftp://$USER:$PASS@$HOST"
LCD="/path/of/your/local/dir"
RCD="/path/of/your/remote/dir"
#DELETE="--delete"
lftp -c "set ftp:list-options -a;
open '$FTPURL';
lcd $LCD;
cd $RCD;
mirror --reverse \
       $DELETE \
       --verbose \
       --exclude-glob a-dir-to-exclude/ \
       --exclude-glob a-file-to-exclude \
       --exclude-glob a-file-group-to-exclude* \
       --exclude-glob other-files-to-exclude"

Warning: make sure that the target directory exists, otherwise the cd command will fail, so operation including deleting trees of files will take place at wrong directory (root)!

I have updated script so that --delete option is disabled by defaut, enable it by uncommenting the DELETE= command.

  • Helped me a great bunch. But I had to remove the "set ftp:list-options -a;" which where messing stuff up. – mat Nov 16 '10 at 20:33
  • 1
    Is there something like this that does the same thing over SFTP (FTPS?)? – trusktr Mar 14 '12 at 2:52
  • 1
    I would do this before cd $RCD, so as to prevent a-script-with-a-typo from accidentally deleting files from the wrong directory: set cmd:fail-exit yes. (It seems lftp doesn't notice that it has actually exited, until you issue the next command.) — And before cd $LCD I'd do this (in the local shell, that is, at the top of the scrip): set -eu. – KajMagnus Jun 28 '12 at 21:05
  • 3
    Actually, the first time I run my script, I had not yet created the remote directory, so the script failed to CD into that non-existing dir. Then the script started to mirror, and thus --delete:d files from the wrong dir. (But I copied them back again, so in that case it didn't matter. Perhaps it matters for someone else.) – KajMagnus Jun 28 '12 at 22:09
  • 1
    As well as following @KajMagnus' advice about using 'set -eu' and 'cd $LCD', add 'mkdir -p $LCD' before 'cd $LCD' - that way you can be sure that the directory exists and you're in it. – Paul Gear Mar 19 '13 at 22:04
25

Depending of what you're actually trying to do, another completely different approach could be use curlftps to mount a ftp folder, and then maybe rsync the two "local" folders.

The installation is different depending on your distro so I can't generalize on that, but you need to installfuse and curlftpfs (prolly Debian already has fuse install by default), then:

  1. sudo apt-get install curlftpfs

  2. Make sure the fuse module is loaded (modprobe fuse)

  3. sudo curlftpfs ftp.yourserver.com /path/to/ftp/folder/ -o user=username:password,allow_other

Note that there's no space after the last comma (it's not a typo!). If you're satisfied with that or don't want to make that every time, you can add it to your fstab (usually in /etc/fstab):

curlftpfs#user:password@ftp.yourserver.com /path/to/ftp/folder/ fuse auto,user,uid=1000,allow_other 0 0

In that case, you have to make sure the fuse module is loaded before.

Be advised though, of two things:

  • That the developer dropped the project one or two years ago, so I don't know how usable/stable for the time being.
  • If the password contains a weird character curlftpfs could fail (maybe you can do something with the .netrc).
  • FTP Fuse and rsync is closest to the question but clearly insane on anything but a small set of small files. – Steve-o Sep 23 '11 at 21:03
  • 1
    Why would it be insane? By default rsync stat()s, it doesn't compare the contents. That can be done by ls -l with an ftp client. – ptman Dec 30 '11 at 11:21
  • I actually tried this (via rsnapshot) and it didn't work, each file returned and error, showing that rsync couldn't create a link. So using lftp in conjunction with rsnapshot did the trick – Konstantin Pereiaslov Feb 11 '13 at 4:25
  • Flawless on 14.04 LTS; an additional tip: when you're ready to unmount your FTP site simply use umount DIR where DIR was your /path/to/ftp/folder above. – Joe Aug 4 '15 at 15:42
  • 1
    2017, and this still works! Honestly the easiest way to copy files from a FTP site enmasse. Mount locally, then rsync as if it were local – td512 May 11 '17 at 5:45
16

There is weex...

Weex is an utility designed to automate the task of remotely maintaining a web page or other FTP archive. It will synchronize a set of local files to a remote server by performing uploads and remote deletes as required.

9

rsync does not work over ftp. On the remote side it needs either the rsync daemon or a shell that it can call rsync from. Ftp generally allows you to call a few commands and rsync is not one of them. There are other tools meant for automating ftp tasks like "lftp".

8

This seems like a good and free fit: https://savannah.gnu.org/projects/ftpsync/

  • 1
    Given the steaming pile of don't-touch-with-a-ten-foot-pole SourceForge has become lately, perhaps you should update the link to the project's webpage. – HopelessN00b Feb 22 '16 at 0:32
7

For what it appears you are trying to do, you could also use wget.

wget -m ftp://username:password@domain.com

  • That will only make a mirror of the site. You can't use this to upload your modifications. – David V. Feb 2 '11 at 10:06
5

You can use curlftpfs to mount ftp and use rsync after http://linux.die.net/man/1/curlftpfs

  • Welcome to ServerFault! This may be a good start, but some more technical detail as to why this is a better approach, and possibly a mock up on how to do it, might be reasonable. – Scott Pack Dec 9 '12 at 22:11
4

I am using GVFS/FTP. I mount my remote FTP servers with gigolo. As they are seen as local directories almost anything working on local files works. rsync is designed to compute file hashes remotely to compare files without transferring them, but doing that with virtual files transfers the files anyway. unison and freefilesync normally work well but I met a problem when they want to rename a file thy uploaded, no problem downloading. This could be a problem with gvfs 1.6.1.

3

Jonas S's solution can be useful depending on the circumstance, for example if you have a high download speed and slow upload, checking the files on the server might be relatively faster than uploading files unnecessarily.

You probably want to use CurlFTPFS, though: http://packages.ubuntu.com/search?suite=default&section=all&arch=any&searchon=names&keywords=curlftpfs

Although, generally, it is better to use a regular FTP client altogether instead of rsync.

2

I don't believe you will be able to do this, the server you are trying to Rsync to will only have an FTP server running, it will not understand the commands that Rsync is sending it.

If the reason you want to do this is that you only have access to port 21, but you have control of the server, you can change the port Rsync listens on, on the server, but this is obviously only useful if you don't want to use FTP on that port.

2

If you want to automate the task then use lftp as you can create a script as some people have posted above, you can script it all really easily to your liking, if your looking for a one time solution (i.e. you need download an entire website via ftp / move it to another server) use ncftp, its simple, install it if its not already installed:

apt-get install ncftp

or

yum install ncftp

(sorry non debian/red hat based distros..)

once installed, open ncftp in the terminal then type:

open -u ftpusername ftp.thedomain.com

It'll ask you for the password, enter it, and then:

get -R /home/remotewebsite/public_html/ /home/localwebsite/public_html/

It'll do the rest pf the work for you.. good luck

1

Using “sitecopy” or “mirror” can save you lots of bandwidth.

They both handle efficient incremental update.

0

There's also wput:

wput --timestamping --reupload --dont-continue source-files ftp://target.host/dir

The username and password are read from ~/.netrc. Otherwise, if you're not particularly concerned with security, you may supply them as ftp://username:password@target.host.

0

I made own command-line tool for that: ftpsync

Why? Because it's simple and elegant, other tools require a lot of dependencies, sometimes compiling, sometimes doesn't work (e.g. lftp on MacOs High Sierra). Ftpsync is written in pure Python, doesn't require 3rd party libs and it's small, you can just include the file in your project.

Usage: cd to your project directory and type:

ftpsync user:pass@ftp.address.url/remote/dir --delete

--delete stands for deleting remote files which don't exist locally.

If you don't want to sync but just run raw ftp command use:

ftpsync -c '...'

It's useful for some simple operations like sending file and rename it:

ftpsync -c 'PASV; TYPE I; STOR some_file.ext; RNFR some_file.ext; RNTO some_file_renamed.ext'
  • Stumbled upon this due to same reason as others do. A need for a good sync utility over FTP protocol. Your tool looks good, but I couldn't put it to use, since my requirement is to download from remote and sync with local. I want to avoid downloading files that are already downloaded. On local I do a processing on downloaded files which makes it differ; that should be skipped next time FTP script runs. At the moment hard time finding a solution. – anup Mar 16 '18 at 13:11
  • Yes, actually it's one-way sync. – superdario128 Mar 19 '18 at 22:48
0

rclone is a nice piece of software that does directory synchronization over ftp and many other protocols including cloud storage providers (Amazon S3, Dropbox, Google Drive). It can synchronize to/from any combination of local or remote storage.

To setup an ftp remote follow these instructions.

Once that is done, the sync operation is:

rclone sync /home/dir FTP_REMOTE:/home --progress 
-7

Install sftp and then you can use rsync.

I use the following one.

rsync --delete --times --recursive --perms --owner --group --verbose --progress --stats -e ssh root@192.168.0.100:/folder1/ /folder2/
  • 7
    That is not FTP it probably uses the sftp subsystem internal to SSH (at least that's what it does in the default config) – serverhorror Jun 12 '09 at 12:27
  • csync(sftp) and rsync may be the same way of working – Rajat Jun 12 '09 at 12:34

protected by Tom O'Connor Jul 20 '13 at 21:36

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.