1

I am trying to use wget to download from my AWS ubuntu server with apache2. I have tried several different options but they all result in either a file by the directory name, or an index.html file.

There are 3 pictures and an ogg format video in the directory.

This is the wget I am using:

wget -N -np http://domainorIP/dir/dir2/dirtodownload/

  • -N is only download files that are newer
  • -np do not ascend to parent directory

3 Answers 3

1

By default, wget only downloads the URL you give it, not any resources linked from an HTML page. To follow these links, you need the -r (or --recursive) option.

There is also the --page-requisites option which downloads any resources necessary for displaying the page you give it, such as inline images and stylesheets.

There are also "grouped" options such as --mirror which include -r along with several other options that are useful for mirroring.

The man page is enormous because wget has a lot of features. The section on Recursive Retrieval Options would be well worth your time to read.

0

Download a CLI browser and download the desired files, you can use ELinks or W3M

apt-get install elinks w3m
0

I've never had success in getting either curl or wget to download files which are being served from an Apache server where indexing directories is enabled. I'm thinking this is your problem as well. The directories show up like this when you browse them:

image of Apache indexed directory

In Apache they're enabled like this for example:

<Directory /var/www/domain.com/pdfs>
Options Indexes FollowSymLinks
</Directory>

Using the shell you're options are limited to getting the list of files and then downloading them one at a time using a command like this one:

% URL="http://www.lamolabs.org/blog/wp-content/uploads/2012/10/"
% curl -s $URL | \
     grep "href" | \
     grep -v "C=D;O=A" | \
     sed "s#^.*href=\"#$URL#" | \
     sed 's/">.*$//' | \
     xargs wget

Breaking this down:

 - URL="..."                - is the URL I want to download files from
 - curl -s $URL             - get's the contents of the index.html generated by Apache
 - grep "href"              - get lines that contain only href
 - grep -v "C=D;O=A"        - eliminate the header bar line generated by Apache
 - sed "s#^.*href=\"#$URL#" - replace .*href=" lines with URL
 - sed 's/">.*$//'          - remove trailing characters >.*$
 - xargs wget               - download each file using wget

You can run this as a single command:

url="http://www.lamolabs.org/blog/wp-content/uploads/2012/10/"; curl -s $url | grep "href" | grep -v "C=D;O=A" | sed "s#^.*href=\"#$url#" | sed 's/">.*$//' | xargs wget 

Running it results in the following files getting downloaded:

% ls -l
total 1652
-rw-rw-r-- 1 saml saml 1351400 Oct  8 23:35 Anatomy-of-the-Linux-file-system.mht
-rw-rw-r-- 1 saml saml     485 Oct 11 00:42 eratosthenes_prime_sieve.pl_.txt
-rw-rw-r-- 1 saml saml   27191 Oct  3 21:42 Selection_005-150x150.png
-rw-rw-r-- 1 saml saml   24202 Oct  3 21:42 Selection_005.png
-rw-rw-r-- 1 saml saml   27141 Oct  3 21:42 Selection_006-150x150.png
-rw-rw-r-- 1 saml saml   24906 Oct  3 21:42 Selection_006.png
-rw-rw-r-- 1 saml saml   25783 Oct  3 22:17 Selection_007-150x150.png
-rw-rw-r-- 1 saml saml  111915 Oct  3 22:17 Selection_007-650x180.png
-rw-rw-r-- 1 saml saml   48109 Oct  4 09:57 Selection_007-e1349359020755.png
-rw-rw-r-- 1 saml saml   29336 Oct  3 22:17 Selection_007.png
1
  • Although I have the index file turned on at this point it has created some problems and must now be turned off.
    – mrhobbeys
    Feb 28, 2013 at 6:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .