One month ago, I used "wget --mirror" to create a mirror of our public website for temporary use during an upcoming scheduled maintenance window. Our primary website runs HTML, PHP & MySQL, but the mirror just needs to be HTML-only, no dynamic-content, PHP or database needed.
The following command will create a simple, online mirror of our website:
wget --mirror http://www.example.org/
Note that the Wget manual says
--mirror "is currently equivalent to
-r -N -l inf --no-remove-listing" (The human-readable equivalent is `--recursive --timestamping --level=inf --no-remove-listing.
Now it's a month later, and much of the website content has changed. I want wget to check all the pages, and download any pages which have changed. However, this isn't working.
What do I need to do to refresh the mirror of the website, short of deleting the directory and re-running the mirror?
The top level file at http://www.example.org/index.html has not changed, but there are many other files which have changed.
I thought all I needed to do was to re-run
wget --mirror, because
--mirror implies the flags
--recursive "specify recursive download" and
--timestamping "Don't re-retrieve files unless newer than local." I thought this would check all of the pages and only retrieve files which are newer then my local copies. Am I wrong?
However, wget doesn't recurse the site on the second try. 'wget --mirror' will check http://www.example.org/index.html , notice that this page did not change, and then stop.
--2010-06-29 10:14:07-- http://www.example.org/ Resolving www.example.org (www.example.org)... 10.10.6.100 Connecting to www.example.org (www.example.org)|10.10.6.100|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Server file no newer than local file "www.example.org/index.html" -- not retrieving. Loading robots.txt; please ignore errors. --2010-06-29 10:14:08-- http://www.example.org/robots.txt Connecting to www.example.org (www.example.org)|10.10.6.100|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 136 [text/plain] Saving to: “www.example.org/robots.txt” 0K 100% 6.48M=0s 2010-06-29 10:14:08 (6.48 MB/s) - "www.example.org/robots.txt" saved [136/136] --2010-06-29 10:14:08-- http://www.example.org/news/gallery/image-01.gif Reusing existing connection to www.example.org:80. HTTP request sent, awaiting response... 200 OK Length: 40741 (40K) [image/gif] Server file no newer than local file "www.example.org/news/gallery/image-01.gif" -- not retrieving. FINISHED --2010-06-29 10:14:08-- Downloaded: 1 files, 136 in 0s (6.48 MB/s)