One month ago, I used "wget --mirror" to create a mirror of our public website for temporary use during an upcoming scheduled maintenance window. Our primary website runs HTML, PHP & MySQL, but the mirror just needs to be HTML-only, no dynamic-content, PHP or database needed.

The following command will create a simple, online mirror of our website:

wget --mirror http://www.example.org/

Note that the Wget manual says --mirror "is currently equivalent to -r -N -l inf --no-remove-listing" (The human-readable equivalent is `--recursive --timestamping --level=inf --no-remove-listing.

Now it's a month later, and much of the website content has changed. I want wget to check all the pages, and download any pages which have changed. However, this isn't working.

My question:

What do I need to do to refresh the mirror of the website, short of deleting the directory and re-running the mirror?

The top level file at http://www.example.org/index.html has not changed, but there are many other files which have changed.

I thought all I needed to do was to re-run wget --mirror, because --mirror implies the flags --recursive "specify recursive download" and --timestamping "Don't re-retrieve files unless newer than local." I thought this would check all of the pages and only retrieve files which are newer then my local copies. Am I wrong?

However, wget doesn't recurse the site on the second try. 'wget --mirror' will check http://www.example.org/index.html , notice that this page did not change, and then stop.

--2010-06-29 10:14:07--  http://www.example.org/
Resolving www.example.org (www.example.org)...
Connecting to www.example.org (www.example.org)||:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Server file no newer than local file "www.example.org/index.html" -- not retrieving.

Loading robots.txt; please ignore errors.
--2010-06-29 10:14:08--  http://www.example.org/robots.txt
Connecting to www.example.org (www.example.org)||:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 136 [text/plain]
Saving to: “www.example.org/robots.txt”

     0K                                                       100% 6.48M=0s
2010-06-29 10:14:08 (6.48 MB/s) - "www.example.org/robots.txt" saved [136/136]

--2010-06-29 10:14:08--  http://www.example.org/news/gallery/image-01.gif
Reusing existing connection to www.example.org:80.
HTTP request sent, awaiting response... 200 OK
Length: 40741 (40K) [image/gif]
Server file no newer than local file "www.example.org/news/gallery/image-01.gif" -- not retrieving.

FINISHED --2010-06-29 10:14:08--
Downloaded: 1 files, 136 in 0s (6.48 MB/s)
  • depending what you are mirroring, you might be able to use rsync instead of wget. Typically available from big archive sites. – Dan Pritts Dec 5 '16 at 21:18

The following workaround seems to work for now. It forcibly deletes /index.html , which forces wget to check all child links again. However, shouldn't wget check all child links automatically?

rm www.example.org/index.html && wget --mirror http://www.example.org/
  • This did the trick for me too. It is cycling through the files with 304 for most.. but downloading some.. (I hope all that has changed) – Bhikkhu Subhuti Aug 13 '19 at 1:47
wget –mirror –w 3 –p –P c:\wget_files\example2 ftp://username:password@ftp.yourdomain.com

This is how I do it on a Windows-based machine http://www.devarticles.com/c/a/Web-Services/Website-Mirroring-With-wget/1/

You can change the path to your directory structure, try downloading all content via FTP and see if that helps.

I also use another utility on Windows "AllwaySync" works superbly.

  • This is a HTTP webserver, running HTML, PHP & MySQL. I am mirroring the site as flat HTML (No PHP or MySQL are needed on the mirror). There is no FTP access to this box. – Stefan Lasiewski Jul 13 '10 at 21:00

I use the --mirror switch to do exactly what you are asking about, which does indeed cause wget to only download newer files recursively. Specifically, my command line (sanitized) is:

/usr/bin/wget -v --mirror ftp://user:password@site/ -o /var/log/webmirror -P /var/WebSites
  • Seems that my wget is getting hung-up on the the fact that 'index.html' doesn't change. Your situation may be a little different, because there is no equivalent to my index.html file. Correct? – Stefan Lasiewski Jun 29 '10 at 17:28
  • 1
    @Stefan, I've just realised that I'm using wget via FTP, not HTTP. That is no doubt why mine is behaving differently. It's been a while since I set this up but I have a vague recollection of having issues when using HTTP. – John Gardeniers Jun 29 '10 at 23:17

You can try to use:

wget -r -l inf -N http://www.example.org/
  • That results in the same behavior. Those options are already implied by -m or --mirror. The manual says is currently equivalent to -r -N -l inf --no-remove-listing. – Stefan Lasiewski Jul 8 '10 at 19:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.