118

I want to copy all of the files and folders from one host to another. The files on the old host sit at /var/www/html and I only have FTP access to that server, and I can't TAR all the files. Regular connection to the old host through FTP brings me to the /home/admin folder.

I tried running the following command form my new server:

wget -r ftp://username:[email protected]

But all I get is a made up index.html file.

What the right syntax for using wget recursively over FTP?

2
  • 5
    Note that, by default, wget -r has a maximum recursion depth of 5; if you have an deeper subdirectories, they will be ignored unless you change this (see -l in the man page).
    – mikewaters
    May 23, 2011 at 19:21
  • What wget does also depends on wget version (or build) I came across one GNU Wget 1.10.2 (Red Hat modified) where when path ends without '/' wget considers it a file and a download fails, but when --timestamping (-N) is used, it would work. Then when path ended in '/' it would always create an index.html instead of downloading files. Ending in * would also work for a dir. In all cases also --recursive was used.
    – papo
    Feb 18, 2022 at 21:22

13 Answers 13

156

Try -m for --mirror

wget -m ftp://username:[email protected]
5
  • 26
    For security reason, it would be good to use --ask-password to avoid password being kept in history.
    – checksum
    Oct 4, 2014 at 2:49
  • This is so cool. Worked like a charm.. Oct 7, 2015 at 21:11
  • This should be the accepted answer (for better visibility).
    – bgondy
    May 3, 2016 at 15:52
  • it doesn't include path of directory to download, I guess @Formoda answer is better
    – Wasim A.
    Apr 24, 2017 at 6:38
  • 1
    @WasimA., you can just add directory path here as well: wget -m ftp://username:[email protected]/mydir/
    – Alex Che
    May 16, 2019 at 15:28
48

You have it right, you just need a trailing * on the end:

wget -r ftp://username:[email protected]/dir/*

For shared servers, you can use like this:

wget -r ftp://1.2.3.4/dir/* --ftp-user=username --ftp-password=password

Because most shared servers has ftp-username something like username@hostname, so, the first wget command not works, and second command works fine.

3
  • 2
    This would only go one directory deep.Better to use the -m flag
    – SvennD
    Aug 13, 2015 at 12:18
  • in some cases second command is not working because of space, You can use first command for share hosting like this. ** wget -r ftp:/ /username%40host:password@host/dir/ **
    – Wasim A.
    Apr 24, 2017 at 7:11
  • 1
    Note: you should use either -m or -r -l inf because -r has a default recursion depth of 5. See gnu.org/software/wget/manual/…
    – ndemou
    Oct 10, 2017 at 20:45
11

Check the below wget command to download data from FTP recursively.

wget --user="<user-name>" --password="<password>" -r -np -nH --cut-dirs=1 --reject "index.html*" "<URL to download files>"

-r: Is for recursively download.

-np: Is for no parent ascending.

-nH: Is for disabling creation of directory having name same as URL i.e. abc.xyz.com

--cut-dirs: Is for ignoring number of parent directories. The value of this option will differ for your command.

You can check by executing the above command.

1
  • What does the --reject do? Is there any way to specify that it is not to overwrite anything?
    – bvargo
    Feb 28, 2023 at 14:01
8

Besides wget, you may also use lftp in script mode. The following command will mirror the content of a given remote FTP directory into the given local directory, and it can be put into the cron job:

lftp -c 'open <hostname>; user <username> <password>; mirror -e <remote-src-path> <local-dest-path>; quit'

It automatically handles recursion into directories and allows specifying the remote source starting directory from to download data from.

5

You can use 'curlftpfs - mount a ftp host as a local directory' and, once mounted, you can use normal file tools like 'cp -r'.

1
  • curlftpfs has major performance problems right now unless you downgrade libcurl3-gnutls
    – Hengjie
    Feb 6, 2015 at 5:40
4

Should work try:

wget -r ftp://ftp:[email protected]/tst/
3

Use -m rather than -r, because of the recursion depth.

http://www.editcorp.com/Personal/Lars_Appel/wget/wget_2.html#SEC11

3

Use:

wget -m ftp://192.168.0.1

and it will mirror all the files and folders.

3
  • As I said, logging in through FTP takes me to the /home/admin folder, and the files I need are in /var/www/html So when I run the command you suggested, I only get the contents of /home/admin I tried running it with 192.168.0.1/var/www/html, but then it tries to CWD /home/admin/var/www/html. How do I make it go to this folder from the root?
    – user9406
    Jun 13, 2009 at 17:23
  • I have no experience with this particular problem, but you could try making a symlink to /var/www/html under your home. Then you could use an address like 192.168.0.1/html Jun 13, 2009 at 17:29
  • I tried creating a symlink, but it resulted with wget creating a similar symlink on my local.
    – user9406
    Jun 13, 2009 at 17:32
1

That's the right syntax. Not sure why you aren't getting the expected results.

As ever there is more than one way to do it. Try ncftp, in particular ncftpget

1
  • I can't install more software on my new server. Shouldn't I be telling wget to download all files from /var/www/html? I tried wget -r username:[email protected]/var/www/html but I got a directory not found error.
    – user9406
    Jun 13, 2009 at 9:37
1

I can understand if you're trying to dump this into cron or something, but why not simply ftp into the server with your normal client and mget *? This might be a quicker path to success.

3
  • I tried mget * but it didn't work with sub-folders, saying the local sub-folder doesn't exist. Is there a way to make him create the local folders automatically?
    – user9406
    Jun 13, 2009 at 17:16
  • Depending on your client, the -r switch usually does the trick. IE: mget -r *
    – dr.pooter
    Jun 15, 2009 at 6:17
  • mmm. AFAIK standard ftp client in linux is not designed to retrieve directories recursively. I mean - there is no -r option. other clients like ncftp or lftp support recursive retrieval but they usually not available by default.
    – Stann
    Feb 13, 2011 at 5:42
1

As I said, logging in through FTP takes me to the /home/admin folder, and the files I need are in /var/www/html

I think this will work in your case:

wget -r ftp://192.168.0.1/../../var/www/html
0

I came across Windows hosting with the username as brinkster/username, so wget would throw an error if you use the syntax:

wget -m ftp://brinkster/username:[email protected]

To get past this, use

wget -m ftp://brinkster%2Fusername:[email protected]

I had to face this with Windows hosting @ brinkster.

-3

wget --user username --password yourpassword ftp://example.com/ftpfiles/filename will do the job or you can add * instead of filename entring : makes wget think of port which is invalid in your case.

1
  • This answer to a four year old question with multiple upvoted answers doesn't offer anything new and even omits the requirement for a recursive copy.
    – Sven
    Jun 20, 2013 at 0:10

You must log in to answer this question.