Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I got a problem with wget, I need to download an entire site with images and other files linked in main pages, I'm using these options:

wget --load-cookies /tmp/cookie.txt -r -l 1 -k -p -nc 'https://www.site.com/mainpage.do'

(-l 1 is used for testing, I may need to travel to level 3 or even 4)

The problem is: I can't figure out how to bypass the 'random' GET parameter that is added after some recursion cycles, so my final result in the /tmp folder is like this:

/tmp/www.site.com/mainpage.do
/tmp/www.site.com/mainpage.do?cx=0.0340590343408
/tmp/www.site.com/mainpage.do?cx=0.0348934786475
/tmp/www.site.com/mainpage.do?cx=0.0032878284787
/tmp/www.site.com/mainpage.do?cx=0.0266389459023
/tmp/www.site.com/mainpage.do?cx=0.0103290334732
/tmp/www.site.com/mainpage.do?cx=0.0890345378478

Since the page it is always the same I don't need to get it other times, I tried with -nc option but it doesn't work, I also tried using -R (reject) but it only works with file extensions, not with URL parameters.

I looked extensively in the wget manual but I don't seem to find a way to do it; it is not mandatory to use wget, if you know how to do it in other ways, they are welcome.

Thankyou

share|improve this question
    
WHat are your purposes? The way I see it you've run into the problem of leeching from a site with anti-leeching code. Perhaps you might also tell us how this qualifies as a system administration question. –  John Gardeniers Oct 23 '09 at 9:31
    
Actually the site I'm trying to download is that of my own employer, and yes, it has security constraints, I'm allowed to do this, we need a static working copy of the site. –  Lex Oct 23 '09 at 12:33
    
I don't know if reallt qualifies as a sysadmin question, I thought it didn't belong to stack overflow and thought about asking it here, sorry if I was wrong, just move it to stackoverflow. –  Lex Oct 23 '09 at 12:34
    
It may become a stack overflow question if you take a programmatic route. Otherwise it's probably a SuperUser question. –  Kyle Smith Oct 23 '09 at 14:09

1 Answer 1

up vote 0 down vote accepted

Write a local proxy server that modifies the responses sent to wget.

Assuming your URLs are in links such as:

<a href="/path/to/mainpage.do?cx=0.0123412341234">

Then you can run a Ruby proxy server like this:

require 'webrick/httpproxy'
s = WEBrick::HTTPProxyServer.new(
   :Port => 2200,
   :ProxyContentHandler => Proc.new{|req,res|
      res.body.gsub!(/mainpage.do?cz=[0-9\.]*/, "mainpage.do")
   } 
)  
trap("INT"){ s.shutdown }
s.start
share|improve this answer
    
I thought about the same thing during the course of the day, I'll implement it with python though, saw this: stackoverflow.com/questions/989739/… –  Lex Oct 23 '09 at 16:22

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.