Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

One month ago, I used "wget --mirror" to create a mirror of our public website for temporary use during an upcoming scheduled maintenance window. Our primary website runs HTML, PHP & MySQL, but the mirror just needs to be HTML-only, no dynamic-content, PHP or database needed.

The following command will create a simple, online mirror of our website:

wget --mirror

Note that the Wget manual says --mirror "is currently equivalent to -r -N -l inf --no-remove-listing" (The human-readable equivalent is `--recursive --timestamping --level=inf --no-remove-listing.

Now it's a month later, and much of the website content has changed. I want wget to check all the pages, and download any pages which have changed. However, this isn't working.

My question:

What do I need to do to refresh the mirror of the website, short of deleting the directory and re-running the mirror?

The top level file at has not changed, but there are many other files which have changed.

I thought all I needed to do was to re-run wget --mirror, because --mirror implies the flags --recursive "specify recursive download" and --timestamping "Don't re-retrieve files unless newer than local." I thought this would check all of the pages and only retrieve files which are newer then my local copies. Am I wrong?

However, wget doesn't recurse the site on the second try. 'wget --mirror' will check , notice that this page did not change, and then stop.

--2010-06-29 10:14:07--
Resolving (
Connecting to (||:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Server file no newer than local file "" -- not retrieving.

Loading robots.txt; please ignore errors.
--2010-06-29 10:14:08--
Connecting to (||:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 136 [text/plain]
Saving to: “”

     0K                                                       100% 6.48M=0s
2010-06-29 10:14:08 (6.48 MB/s) - "" saved [136/136]

--2010-06-29 10:14:08--
Reusing existing connection to
HTTP request sent, awaiting response... 200 OK
Length: 40741 (40K) [image/gif]
Server file no newer than local file "" -- not retrieving.

FINISHED --2010-06-29 10:14:08--
Downloaded: 1 files, 136 in 0s (6.48 MB/s)
share|improve this question
up vote 1 down vote accepted

The following workaround seems to work for now. It forcibly deletes /index.html , which forces wget to check all child links again. However, shouldn't wget check all child links automatically?

rm && wget --mirror
share|improve this answer

wget –mirror –w 3 –p –P c:\wget_files\example2

this is how I do it on windows based machine

You can change the path to your directory structure, try downloading all content via ftp and see if that helps.

I also use another utility on Windows "AllwaySync" works superb.

share|improve this answer
This is a HTTP webserver, running HTML, PHP & MySQL. I am mirroring the site as flat HTML (No PHP or MySQL are needed on the mirror). There is no FTP access to this box. – Stefan Lasiewski Jul 13 '10 at 21:00

I use the --mirror switch to do exactly what you are asking about, which does indeed cause wget to only download newer files recursively. Specifically, my command line (sanitized) is:

/usr/bin/wget -v --mirror ftp://user:password@site/ -o /var/log/webmirror -P /var/WebSites
share|improve this answer
Seems that my wget is getting hung-up on the the fact that 'index.html' doesn't change. Your situation may be a little different, because there is no equivalent to my index.html file. Correct? – Stefan Lasiewski Jun 29 '10 at 17:28
@Stefan, I've just realised that I'm using wget via FTP, not HTTP. That is no doubt why mine is behaving differently. It's been a while since I set this up but I have a vague recollection of having issues when using HTTP. – John Gardeniers Jun 29 '10 at 23:17

You can try to use:

wget -r -l inf -N
share|improve this answer
That results in the same behavior. Those options are already implied by -m or --mirror. The manual says is currently equivalent to -r -N -l inf --no-remove-listing. – Stefan Lasiewski Jul 8 '10 at 19:47

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.