Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

One month ago, I used "wget --mirror" to create a mirror of our public website for temporary use during an upcoming scheduled maintenance window. Our primary website runs HTML, PHP & MySQL, but the mirror just needs to be HTML-only, no dynamic-content, PHP or database needed.

The following command will create a simple, online mirror of our website:

wget --mirror http://www.example.org/

Note that the Wget manual says --mirror "is currently equivalent to -r -N -l inf --no-remove-listing" (The human-readable equivalent is `--recursive --timestamping --level=inf --no-remove-listing.

Now it's a month later, and much of the website content has changed. I want wget to check all the pages, and download any pages which have changed. However, this isn't working.

My question:

What do I need to do to refresh the mirror of the website, short of deleting the directory and re-running the mirror?

The top level file at http://www.example.org/index.html has not changed, but there are many other files which have changed.

I thought all I needed to do was to re-run wget --mirror, because --mirror implies the flags --recursive "specify recursive download" and --timestamping "Don't re-retrieve files unless newer than local." I thought this would check all of the pages and only retrieve files which are newer then my local copies. Am I wrong?

However, wget doesn't recurse the site on the second try. 'wget --mirror' will check http://www.example.org/index.html , notice that this page did not change, and then stop.

--2010-06-29 10:14:07--  http://www.example.org/
Resolving www.example.org (www.example.org)... 10.10.6.100
Connecting to www.example.org (www.example.org)|10.10.6.100|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Server file no newer than local file "www.example.org/index.html" -- not retrieving.

Loading robots.txt; please ignore errors.
--2010-06-29 10:14:08--  http://www.example.org/robots.txt
Connecting to www.example.org (www.example.org)|10.10.6.100|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 136 [text/plain]
Saving to: “www.example.org/robots.txt”

     0K                                                       100% 6.48M=0s
2010-06-29 10:14:08 (6.48 MB/s) - "www.example.org/robots.txt" saved [136/136]

--2010-06-29 10:14:08--  http://www.example.org/news/gallery/image-01.gif
Reusing existing connection to www.example.org:80.
HTTP request sent, awaiting response... 200 OK
Length: 40741 (40K) [image/gif]
Server file no newer than local file "www.example.org/news/gallery/image-01.gif" -- not retrieving.

FINISHED --2010-06-29 10:14:08--
Downloaded: 1 files, 136 in 0s (6.48 MB/s)
share|improve this question
add comment

4 Answers

up vote 1 down vote accepted

The following workaround seems to work for now. It forcibly deletes /index.html , which forces wget to check all child links again. However, shouldn't wget check all child links automatically?

rm www.example.org/index.html && wget --mirror http://www.example.org/
share|improve this answer
add comment

wget –mirror –w 3 –p –P c:\wget_files\example2 ftp://username:password@ftp.yourdomain.com

this is how I do it on windows based machine http://www.devarticles.com/c/a/Web-Services/Website-Mirroring-With-wget/1/

You can change the path to your directory structure, try downloading all content via ftp and see if that helps.

I also use another utility on Windows "AllwaySync" works superb.

share|improve this answer
    
This is a HTTP webserver, running HTML, PHP & MySQL. I am mirroring the site as flat HTML (No PHP or MySQL are needed on the mirror). There is no FTP access to this box. –  Stefan Lasiewski Jul 13 '10 at 21:00
add comment

I use the --mirror switch to do exactly what you are asking about, which does indeed cause wget to only download newer files recursively. Specifically, my command line (sanitized) is:

/usr/bin/wget -v --mirror ftp://user:password@site/ -o /var/log/webmirror -P /var/WebSites
share|improve this answer
    
Seems that my wget is getting hung-up on the the fact that 'index.html' doesn't change. Your situation may be a little different, because there is no equivalent to my index.html file. Correct? –  Stefan Lasiewski Jun 29 '10 at 17:28
1  
@Stefan, I've just realised that I'm using wget via FTP, not HTTP. That is no doubt why mine is behaving differently. It's been a while since I set this up but I have a vague recollection of having issues when using HTTP. –  John Gardeniers Jun 29 '10 at 23:17
add comment

You can try to use:

wget -r -l inf -N http://www.example.org/
share|improve this answer
    
That results in the same behavior. Those options are already implied by -m or --mirror. The manual says is currently equivalent to -r -N -l inf --no-remove-listing. –  Stefan Lasiewski Jul 8 '10 at 19:47
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.