1

Is it possible to wget an entire index/parent directory with wget? Because I am trying to transfer images to a VPS of mine from my old shared host.

When I say an index/parent directory, I meant like this:

enter image description here

So basically I want wget to go through each file and download it until all have been downloaded.

Cheers.

6
  • just playing devils advocate here...you don't have ftp access to your old shared host?
    – iainlbc
    May 1 '11 at 22:09
  • I agree with @iain, if its your vps can't you just tar/zip or transfer using ftp/sftp/scp?
    – coredump
    May 1 '11 at 22:14
  • I do have ftp access, but I don't know how to do FTP from vps to old host.
    – MacMac
    May 1 '11 at 22:14
  • You can use your desktop as the middleman- Download everything from old server to your desktop, FTP to new server, upload.
    – iainlbc
    May 1 '11 at 22:20
  • Not with my 100kb/s speeds, no. That would take me forever lol.
    – MacMac
    May 1 '11 at 22:21
2

you did mention there is ftp access, wget actually supports ftp out of the box:

wget -r -l0 --ftp-user=user --ftp-password=pass ftp://yourdomain.com/dir/*
0
2

the -r (recursive) wget switch should work, -l X specifies how deep to traverse (5 levels max):

wget -r -l 1 http://domain.com/directory/

source: http://www.gnu.org/software/wget/manual/wget.html#Recursive-Download

1
  • Damn! I knew there was an easier way. Mine's more creative. :) Nicely done!
    – jdcarg
    May 1 '11 at 22:32
0

Assuing you have no shell or FTP access nor any control panel, and that this is a legitimate request, you might want to have a look at spidering software - I've found pavuk useful for this in the past.

0

Not sure if wget supports this or not, but you could use a bash script to accomplish this fairly easily.

Step 1. Copy and paste all of the text from the "index of /i" into a file called "files" or something else that makes sense on the new host.

Step 2. On your new machine use the following command to do a search & replace to create the wget commands for each file you need to download.

cat files | sed "s/^/wget http:\/\/yoursite.com\/path\//"

This reads the file named "files" and then pipes it to sed which inserts "wget http://yoursite.com/path/file1.png"

For example I ran it on a file on my local computer. The "files" file contains just the .png files like the following:

blah.png
blah1.png
blah2.png

and the output looks like the following after running the command posted above.

wget http://yoursite.com/path/blah.png
wget http://yoursite.com/path/blah1.png
wget http://yoursite.com/path/blah2.png

At that point, you just copy and paste the all the wgets that are generated in terminal or from command line it it will download each file.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.