wget an remote website index directory

Question

Is it possible to wget an entire index/parent directory with wget? Because I am trying to transfer images to a VPS of mine from my old shared host.

When I say an index/parent directory, I meant like this:

So basically I want wget to go through each file and download it until all have been downloaded.

Cheers.

Answer 1

you did mention there is ftp access, wget actually supports ftp out of the box:

wget -r -l0 --ftp-user=user --ftp-password=pass ftp://yourdomain.com/dir/*

Answer 2

the -r (recursive) wget switch should work, -l X specifies how deep to traverse (5 levels max):

wget -r -l 1 http://domain.com/directory/

source: http://www.gnu.org/software/wget/manual/wget.html#Recursive-Download

Answer 3

Assuing you have no shell or FTP access nor any control panel, and that this is a legitimate request, you might want to have a look at spidering software - I've found pavuk useful for this in the past.

Answer 4

Not sure if wget supports this or not, but you could use a bash script to accomplish this fairly easily.

Step 1. Copy and paste all of the text from the "index of /i" into a file called "files" or something else that makes sense on the new host.

Step 2. On your new machine use the following command to do a search & replace to create the wget commands for each file you need to download.

cat files | sed "s/^/wget http:\/\/yoursite.com\/path\//"

This reads the file named "files" and then pipes it to sed which inserts "wget http://yoursite.com/path/file1.png"

For example I ran it on a file on my local computer. The "files" file contains just the .png files like the following:

blah.png
blah1.png
blah2.png

and the output looks like the following after running the command posted above.

wget http://yoursite.com/path/blah.png
wget http://yoursite.com/path/blah1.png
wget http://yoursite.com/path/blah2.png

At that point, you just copy and paste the all the wgets that are generated in terminal or from command line it it will download each file.