11

One month ago, I used "wget --mirror" to create a mirror of our public website for temporary use during an upcoming scheduled maintenance window. Our primary website runs HTML, PHP & MySQL, but the mirror just needs to be HTML-only, no dynamic-content, PHP or database needed.

The following command will create a simple, online mirror of our website:

wget --mirror http://www.example.org/

Note that the Wget manual says --mirror "is currently equivalent to -r -N -l inf --no-remove-listing" (The human-readable equivalent is `--recursive --timestamping --level=inf --no-remove-listing.

Now it's a month later, and much of the website content has changed. I want wget to check all the pages, and download any pages which have changed. However, this isn't working.

My question:

What do I need to do to refresh the mirror of the website, short of deleting the directory and re-running the mirror?

The top level file at http://www.example.org/index.html has not changed, but there are many other files which have changed.

I thought all I needed to do was to re-run wget --mirror, because --mirror implies the flags --recursive "specify recursive download" and --timestamping "Don't re-retrieve files unless newer than local." I thought this would check all of the pages and only retrieve files which are newer then my local copies. Am I wrong?

However, wget doesn't recurse the site on the second try. 'wget --mirror' will check http://www.example.org/index.html , notice that this page did not change, and then stop.

--2010-06-29 10:14:07--  http://www.example.org/
Resolving www.example.org (www.example.org)... 10.10.6.100
Connecting to www.example.org (www.example.org)|10.10.6.100|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Server file no newer than local file "www.example.org/index.html" -- not retrieving.

Loading robots.txt; please ignore errors.
--2010-06-29 10:14:08--  http://www.example.org/robots.txt
Connecting to www.example.org (www.example.org)|10.10.6.100|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 136 [text/plain]
Saving to: “www.example.org/robots.txt”

     0K                                                       100% 6.48M=0s
2010-06-29 10:14:08 (6.48 MB/s) - "www.example.org/robots.txt" saved [136/136]

--2010-06-29 10:14:08--  http://www.example.org/news/gallery/image-01.gif
Reusing existing connection to www.example.org:80.
HTTP request sent, awaiting response... 200 OK
Length: 40741 (40K) [image/gif]
Server file no newer than local file "www.example.org/news/gallery/image-01.gif" -- not retrieving.

FINISHED --2010-06-29 10:14:08--
Downloaded: 1 files, 136 in 0s (6.48 MB/s)
1
  • depending what you are mirroring, you might be able to use rsync instead of wget. Typically available from big archive sites.
    – Dan Pritts
    Dec 5, 2016 at 21:18

4 Answers 4

7

The following workaround seems to work for now. It forcibly deletes /index.html , which forces wget to check all child links again. However, shouldn't wget check all child links automatically?

rm www.example.org/index.html && wget --mirror http://www.example.org/
1
  • This did the trick for me too. It is cycling through the files with 304 for most.. but downloading some.. (I hope all that has changed) Aug 13, 2019 at 1:47
-1
wget –mirror –w 3 –p –P c:\wget_files\example2 ftp://username:[email protected]

This is how I do it on a Windows-based machine http://www.devarticles.com/c/a/Web-Services/Website-Mirroring-With-wget/1/

You can change the path to your directory structure, try downloading all content via FTP and see if that helps.

I also use another utility on Windows "AllwaySync" works superbly.

1
  • This is a HTTP webserver, running HTML, PHP & MySQL. I am mirroring the site as flat HTML (No PHP or MySQL are needed on the mirror). There is no FTP access to this box. Jul 13, 2010 at 21:00
-2

I use the --mirror switch to do exactly what you are asking about, which does indeed cause wget to only download newer files recursively. Specifically, my command line (sanitized) is:

/usr/bin/wget -v --mirror ftp://user:password@site/ -o /var/log/webmirror -P /var/WebSites
2
  • Seems that my wget is getting hung-up on the the fact that 'index.html' doesn't change. Your situation may be a little different, because there is no equivalent to my index.html file. Correct? Jun 29, 2010 at 17:28
  • 1
    @Stefan, I've just realised that I'm using wget via FTP, not HTTP. That is no doubt why mine is behaving differently. It's been a while since I set this up but I have a vague recollection of having issues when using HTTP. Jun 29, 2010 at 23:17
-2

You can try to use:

wget -r -l inf -N http://www.example.org/
1
  • That results in the same behavior. Those options are already implied by -m or --mirror. The manual says is currently equivalent to -r -N -l inf --no-remove-listing. Jul 8, 2010 at 19:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .