I have an account on a web page that is now "full" (ie I have used up all my allocated space) and I would like to make a mirror of that site. wget seems like the thing to use.

The problem is that I would only like to mirror the sites the lie within this directory http://user.domain.com/room/2324343/transcript/ (and sub-directories). Whilst saving the correct stylesheets, javascripts and css etc which exist in different directories.

There as also uploaded files that are linked to within the pages in the transcript directory (on different directories) that I would like to download/mirror (theses are in a variatey of formats .exe, .py, .png, .app (and many more)). There are also images that are on different severs that are on these pages.

Also I would like it if the links (which are sometimes relative , sometimes absoulute (but to internal things), sometimes external ) worked correctly so that if they link to things that have been downloaded(mirrored) they work fine (without internet connection), but if they link to things that are external or havent been mirrored they link to the external site. Basically so they work as expected.

Another problem is that you have to log in to acess the site.

Can wget be used to acomplish this or is there a better way?

either way how do I achive this?

(I have asked this question at stackoverflow.com/questions/2190115/wget-mirroring-subdomains-and-directories-and-cookies but it was recommended that I try asking it here)


First, you have to log into the server, there are two possibilities, your login gets done throught http auth, in that case, doing something like this :

wget -k -r -m -np --user=foo --password=bar http://user.domain.com/room/2324343/transcript/

should do it.

Now, if you need to log in via a form, you have to do it in two steps, first login, then mirror, so, doing something like :

wget --save-cookies cookies.txt --post-data 'user=foo&pass=bar' http://user.domain.com/room/2324343/transcript/
wget -k -r -m -np --load-cookies cookies.txt http://user.domain.com/room/2324343/transcript/

for the first line, the url may not be right, as the form may send you to a completely different place which would send you back to the right place, and the post-data argument is an example, you'll have to figure out by yourself what to put in it, either via looking at the source of the page, or using some extension for your browser, like firebug for firefox.

  • I will give this a try and report back, thank-you – Jimmu Feb 3 '10 at 8:58

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.