215

Any unix:

I have the following cmd line which works fine.

rsync -avr -e ssh /home/dir [email protected]:/home/

But I need to set it up now to rsync to a remote server that only has an FTP server on it. How do I go about that?

I looked at the rsync help but quickly got lost (I don't do this stuff very often).

6
  • 21
    This is akin to asking how to use HTTP over IRC, or how to use FTP over SMTP...
    – Juliano
    Nov 11, 2009 at 12:26
  • 2
    Maybe you could explain some of your reasoning for wanting to do this, then we can help you come up with a viable solution.
    – James
    Jul 5, 2011 at 21:06
  • 1
    essentially i was trying to automate a backup from a shared hosting server, and didn't understand how rsync worked. i have moved to a vps now, and don't have the limitation of ftp anymore
    – bumperbox
    Jul 5, 2011 at 21:58
  • Duplicity is your god: scie.nti.st/2013/4/13/…
    – user956584
    Feb 27, 2015 at 15:02
  • If deployment is, what's on your mind, and you don't mind NodeJS, check out dploy, lazy and works well with git
    – Frank N
    Dec 12, 2016 at 14:42

18 Answers 18

90

You don't. rsync can't do that for you, it is a protocol of its own and doesn't work over FTP.

You might, however, want to try csync. IIRC it provides rsync-like behaviour over HTTP. I can't comment on whether it works over FTP, you'll have to try it.

1
  • 3
    Looking at the csync web page, it supports sftp but there's no mention of support for plain ftp. Jul 11, 2014 at 2:06
183

rsync isn't going to work for you for the reasons others have mentioned. However, lftp and ncftp both have "mirror" modes that will probably meet your needs.

I use this to push stuff from my local directory to a ftp or sftp web host:

lftp -c "set ftp:list-options -a;
open ftp://user:[email protected]; 
lcd ./web;
cd /web/public_html;
mirror --reverse --delete --use-cache --verbose --allow-chown  
--allow-suid --no-umask --parallel=2 --exclude-glob .svn"
3
  • 3
    lftp is very nice. another one i just found out about is wput which has built-in bandwidth limiting.
    – karmakaze
    Jul 17, 2013 at 6:46
  • I would suggest to add "set ftp:use-mdtm-overloaded true" in order to set file mtime to the same as original (should one uses the other way round, viz. no reverse). May 29, 2020 at 12:43
  • See also stackoverflow.com/a/70058912/9134997 for command line options of lftp. There are really good parameters for syncing data! Nov 21, 2021 at 22:26
133

As written by easel, lftp is a good tool.

I suggest you to parametrize the script, and make use of the

exclude-glob

options, that excludes filenames using the glob feature (*,? ..) of your shell:

#!/bin/bash    
HOST="your.ftp.host.dom"
USER="username"
PASS="password"
FTPURL="ftp://$USER:$PASS@$HOST"
LCD="/path/of/your/local/dir"
RCD="/path/of/your/remote/dir"
#DELETE="--delete"
lftp -c "set ftp:list-options -a;
open '$FTPURL';
lcd $LCD;
cd $RCD;
mirror --reverse \
       $DELETE \
       --verbose \
       --exclude-glob a-dir-to-exclude/ \
       --exclude-glob a-file-to-exclude \
       --exclude-glob a-file-group-to-exclude* \
       --exclude-glob other-files-to-exclude"

Warning: make sure that the target directory exists, otherwise the cd command will fail, so operation including deleting trees of files will take place at wrong directory (root)!

I have updated script so that --delete option is disabled by defaut, enable it by uncommenting the DELETE= command.

11
  • Helped me a great bunch. But I had to remove the "set ftp:list-options -a;" which where messing stuff up.
    – mat
    Nov 16, 2010 at 20:33
  • 1
    Is there something like this that does the same thing over SFTP (FTPS?)?
    – trusktr
    Mar 14, 2012 at 2:52
  • 1
    I would do this before cd $RCD, so as to prevent a-script-with-a-typo from accidentally deleting files from the wrong directory: set cmd:fail-exit yes. (It seems lftp doesn't notice that it has actually exited, until you issue the next command.) — And before cd $LCD I'd do this (in the local shell, that is, at the top of the scrip): set -eu.
    – KajMagnus
    Jun 28, 2012 at 21:05
  • 3
    Actually, the first time I run my script, I had not yet created the remote directory, so the script failed to CD into that non-existing dir. Then the script started to mirror, and thus --delete:d files from the wrong dir. (But I copied them back again, so in that case it didn't matter. Perhaps it matters for someone else.)
    – KajMagnus
    Jun 28, 2012 at 22:09
  • 1
    As well as following @KajMagnus' advice about using 'set -eu' and 'cd $LCD', add 'mkdir -p $LCD' before 'cd $LCD' - that way you can be sure that the directory exists and you're in it.
    – Paul Gear
    Mar 19, 2013 at 22:04
28

Depending of what you're actually trying to do, another completely different approach could be use curlftps to mount a ftp folder, and then maybe rsync the two "local" folders.

The installation is different depending on your distro so I can't generalize on that, but you need to installfuse and curlftpfs (prolly Debian already has fuse install by default), then:

  1. sudo apt-get install curlftpfs

  2. Make sure the fuse module is loaded (modprobe fuse)

  3. sudo curlftpfs ftp.yourserver.com /path/to/ftp/folder/ -o user=username:password,allow_other

Note that there's no space after the last comma (it's not a typo!). If you're satisfied with that or don't want to make that every time, you can add it to your fstab (usually in /etc/fstab):

curlftpfs#user:[email protected] /path/to/ftp/folder/ fuse auto,user,uid=1000,allow_other 0 0

In that case, you have to make sure the fuse module is loaded before.

Be advised though, of two things:

  • That the developer dropped the project one or two years ago, so I don't know how usable/stable for the time being.
  • If the password contains a weird character curlftpfs could fail (maybe you can do something with the .netrc).
7
  • FTP Fuse and rsync is closest to the question but clearly insane on anything but a small set of small files.
    – Steve-o
    Sep 23, 2011 at 21:03
  • 1
    Why would it be insane? By default rsync stat()s, it doesn't compare the contents. That can be done by ls -l with an ftp client.
    – ptman
    Dec 30, 2011 at 11:21
  • I actually tried this (via rsnapshot) and it didn't work, each file returned and error, showing that rsync couldn't create a link. So using lftp in conjunction with rsnapshot did the trick Feb 11, 2013 at 4:25
  • Flawless on 14.04 LTS; an additional tip: when you're ready to unmount your FTP site simply use umount DIR where DIR was your /path/to/ftp/folder above.
    – Joe
    Aug 4, 2015 at 15:42
  • 2
    2017, and this still works! Honestly the easiest way to copy files from a FTP site enmasse. Mount locally, then rsync as if it were local
    – td512
    May 11, 2017 at 5:45
17

There is weex...

Weex is an utility designed to automate the task of remotely maintaining a web page or other FTP archive. It will synchronize a set of local files to a remote server by performing uploads and remote deletes as required.

1
13

For what it appears you are trying to do, you could also use wget.

wget -m ftp://username:[email protected]

1
  • 1
    That will only make a mirror of the site. You can't use this to upload your modifications.
    – David V.
    Feb 2, 2011 at 10:06
12

rsync does not work over ftp. On the remote side it needs either the rsync daemon or a shell that it can call rsync from. Ftp generally allows you to call a few commands and rsync is not one of them. There are other tools meant for automating ftp tasks like "lftp".

8

This seems like a good and free fit: https://savannah.gnu.org/projects/ftpsync/

1
  • 1
    Given the steaming pile of don't-touch-with-a-ten-foot-pole SourceForge has become lately, perhaps you should update the link to the project's webpage. Feb 22, 2016 at 0:32
5

I am using GVFS/FTP. I mount my remote FTP servers with gigolo. As they are seen as local directories almost anything working on local files works. rsync is designed to compute file hashes remotely to compare files without transferring them, but doing that with virtual files transfers the files anyway. unison and freefilesync normally work well but I met a problem when they want to rename a file thy uploaded, no problem downloading. This could be a problem with gvfs 1.6.1.

5

You can use curlftpfs to mount ftp and use rsync after http://linux.die.net/man/1/curlftpfs

1
  • Welcome to ServerFault! This may be a good start, but some more technical detail as to why this is a better approach, and possibly a mock up on how to do it, might be reasonable.
    – Scott Pack
    Dec 9, 2012 at 22:11
4

rclone is a nice piece of software that does directory synchronization over ftp and many other protocols including cloud storage providers (Amazon S3, Dropbox, Google Drive). It can synchronize to/from any combination of local or remote storage.

To setup an ftp remote follow these instructions.

Once that is done, the sync operation is:

rclone sync /home/dir FTP_REMOTE:/home --progress 
3

Jonas S's solution can be useful depending on the circumstance, for example if you have a high download speed and slow upload, checking the files on the server might be relatively faster than uploading files unnecessarily.

You probably want to use CurlFTPFS, though: http://packages.ubuntu.com/search?suite=default&section=all&arch=any&searchon=names&keywords=curlftpfs

Although, generally, it is better to use a regular FTP client altogether instead of rsync.

3

If you want to automate the task then use lftp as you can create a script as some people have posted above, you can script it all really easily to your liking, if your looking for a one time solution (i.e. you need download an entire website via ftp / move it to another server) use ncftp, its simple, install it if its not already installed:

apt-get install ncftp

or

yum install ncftp

(sorry non debian/red hat based distros..)

once installed, open ncftp in the terminal then type:

open -u ftpusername ftp.thedomain.com

It'll ask you for the password, enter it, and then:

get -R /home/remotewebsite/public_html/ /home/localwebsite/public_html/

It'll do the rest pf the work for you.. good luck

2

I don't believe you will be able to do this, the server you are trying to Rsync to will only have an FTP server running, it will not understand the commands that Rsync is sending it.

If the reason you want to do this is that you only have access to port 21, but you have control of the server, you can change the port Rsync listens on, on the server, but this is obviously only useful if you don't want to use FTP on that port.

1

Using “sitecopy” or “mirror” can save you lots of bandwidth.

They both handle efficient incremental update.

1
0

There's also wput:

wput --timestamping --reupload --dont-continue source-files ftp://target.host/dir

The username and password are read from ~/.netrc. Otherwise, if you're not particularly concerned with security, you may supply them as ftp://username:[email protected].

0

I made own command-line tool for that: ftpsync

Why? Because it's simple and elegant, other tools require a lot of dependencies, sometimes compiling, sometimes doesn't work (e.g. lftp on MacOs High Sierra). Ftpsync is written in pure Python, doesn't require 3rd party libs and it's small, you can just include the file in your project.

Usage: cd to your project directory and type:

ftpsync user:[email protected]/remote/dir --delete

--delete stands for deleting remote files which don't exist locally.

If you don't want to sync but just run raw ftp command use:

ftpsync -c '...'

It's useful for some simple operations like sending file and rename it:

ftpsync -c 'PASV; TYPE I; STOR some_file.ext; RNFR some_file.ext; RNTO some_file_renamed.ext'
2
  • Stumbled upon this due to same reason as others do. A need for a good sync utility over FTP protocol. Your tool looks good, but I couldn't put it to use, since my requirement is to download from remote and sync with local. I want to avoid downloading files that are already downloaded. On local I do a processing on downloaded files which makes it differ; that should be skipped next time FTP script runs. At the moment hard time finding a solution.
    – anup
    Mar 16, 2018 at 13:11
  • Yes, actually it's one-way sync. Mar 19, 2018 at 22:48
-7

Install sftp and then you can use rsync.

I use the following one.

rsync --delete --times --recursive --perms --owner --group --verbose --progress --stats -e ssh [email protected]:/folder1/ /folder2/
2
  • 7
    That is not FTP it probably uses the sftp subsystem internal to SSH (at least that's what it does in the default config) Jun 12, 2009 at 12:27
  • csync(sftp) and rsync may be the same way of working
    – Rajat
    Jun 12, 2009 at 12:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .