104

I want to copy all of the files and folders from one host to another. The files on the old host sit at /var/www/html and I only have FTP access to that server, and I can't TAR all the files. Regular connection to the old host through FTP brings me to the /home/admin folder.

I tried running the following command form my new server:

wget -r ftp://username:password@ip.of.old.host

But all I get is a made up index.html file.

What the right syntax for using wget recursively over FTP?

  • 3
    Note that, by default, wget -r has a maximum recursion depth of 5; if you have an deeper subdirectories, they will be ignored unless you change this (see -l in the man page). – mikewaters May 23 '11 at 19:21

13 Answers 13

136

Try -m for --mirror

wget -m ftp://username:password@ip.of.old.host
| improve this answer | |
  • 19
    For security reason, it would be good to use --ask-password to avoid password being kept in history. – checksum Oct 4 '14 at 2:49
  • This is so cool. Worked like a charm.. – Sambhav Sharma Oct 7 '15 at 21:11
  • This should be the accepted answer (for better visibility). – bgondy May 3 '16 at 15:52
  • it doesn't include path of directory to download, I guess @Formoda answer is better – Wasim A. Apr 24 '17 at 6:38
  • @WasimA., you can just add directory path here as well: wget -m ftp://username:password@ip.of.old.host/mydir/ – Alex Che May 16 '19 at 15:28
42

You have it right, you just need a trailing * on the end:

wget -r ftp://username:password@1.2.3.4/dir/*

For shared servers, you can use like this:

wget -r ftp://1.2.3.4/dir/* --ftp-user=username --ftp-password=password

Because most shared servers has ftp-username something like username@hostname, so, the first wget command not works, and second command works fine.

| improve this answer | |
  • 2
    This would only go one directory deep.Better to use the -m flag – SvennD Aug 13 '15 at 12:18
  • in some cases second command is not working because of space, You can use first command for share hosting like this. ** wget -r ftp:/ /username%40host:password@host/dir/ ** – Wasim A. Apr 24 '17 at 7:11
  • 1
    Note: you should use either -m or -r -l inf because -r has a default recursion depth of 5. See gnu.org/software/wget/manual/… – ndemou Oct 10 '17 at 20:45
7

Besides wget, you may also use lftp in script mode. The following command will mirror the content of a given remote FTP directory into the given local directory, and it can be put into the cron job:

lftp -c 'open <hostname>; user <username> <password>; mirror -e <remote-src-path> <local-dest-path>; quit'

It automatically handles recursion into directories and allows specifying the remote source starting directory from to download data from.

| improve this answer | |
5

You can use 'curlftpfs - mount a ftp host as a local directory' and, once mounted, you can use normal file tools like 'cp -r'.

| improve this answer | |
  • curlftpfs has major performance problems right now unless you downgrade libcurl3-gnutls – Hengjie Feb 6 '15 at 5:40
5

Check the below wget command to download data from FTP recursively.

wget --user="<user-name>" --password="<password>" -r -np -nH --cut-dirs=1 --reject "index.html*" "<URL to download files>"

-r: Is for recursively download.

-np: Is for no parent ascending.

-nH: Is for disabling creation of directory having name same as URL i.e. abc.xyz.com

--cut-dirs: Is for ignoring number of parent directories. The value of this option will differ for your command.

You can check by executing the above command.

| improve this answer | |
3

Use -m rather than -r, because of the recursion depth.

http://www.editcorp.com/Personal/Lars_Appel/wget/wget_2.html#SEC11

| improve this answer | |
2

Should work try:

wget -r ftp://ftp:ftp@ftp.sunet.se/tst/
| improve this answer | |
2

Use:

wget -m ftp://192.168.0.1

and it will mirror all the files and folders.

| improve this answer | |
  • As I said, logging in through FTP takes me to the /home/admin folder, and the files I need are in /var/www/html So when I run the command you suggested, I only get the contents of /home/admin I tried running it with 192.168.0.1/var/www/html, but then it tries to CWD /home/admin/var/www/html. How do I make it go to this folder from the root? – user9406 Jun 13 '09 at 17:23
  • I have no experience with this particular problem, but you could try making a symlink to /var/www/html under your home. Then you could use an address like 192.168.0.1/html – prestomation Jun 13 '09 at 17:29
  • I tried creating a symlink, but it resulted with wget creating a similar symlink on my local. – user9406 Jun 13 '09 at 17:32
1

That's the right syntax. Not sure why you aren't getting the expected results.

As ever there is more than one way to do it. Try ncftp, in particular ncftpget

| improve this answer | |
  • I can't install more software on my new server. Shouldn't I be telling wget to download all files from /var/www/html? I tried wget -r username:password@ip.of.old.host/var/www/html but I got a directory not found error. – user9406 Jun 13 '09 at 9:37
1

I can understand if you're trying to dump this into cron or something, but why not simply ftp into the server with your normal client and mget *? This might be a quicker path to success.

| improve this answer | |
  • I tried mget * but it didn't work with sub-folders, saying the local sub-folder doesn't exist. Is there a way to make him create the local folders automatically? – user9406 Jun 13 '09 at 17:16
  • Depending on your client, the -r switch usually does the trick. IE: mget -r * – dr.pooter Jun 15 '09 at 6:17
  • mmm. AFAIK standard ftp client in linux is not designed to retrieve directories recursively. I mean - there is no -r option. other clients like ncftp or lftp support recursive retrieval but they usually not available by default. – Stann Feb 13 '11 at 5:42
1

As I said, logging in through FTP takes me to the /home/admin folder, and the files I need are in /var/www/html

I think this will work in your case:

wget -r ftp://192.168.0.1/../../var/www/html
| improve this answer | |
0

I came across Windows hosting with the username as brinkster/username, so wget would throw an error if you use the syntax:

wget -m ftp://brinkster/username:password@ip.of.the.host

To get past this, use

wget -m ftp://brinkster%2Fusername:password@ip.of.old.host

I had to face this with Windows hosting @ brinkster.

| improve this answer | |
-3

wget --user username --password yourpassword ftp://example.com/ftpfiles/filename will do the job or you can add * instead of filename entring : makes wget think of port which is invalid in your case.

| improve this answer | |
  • This answer to a four year old question with multiple upvoted answers doesn't offer anything new and even omits the requirement for a recursive copy. – Sven Jun 20 '13 at 0:10