Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm trying to download two sites for inclusion on a CD:

http://boinc.berkeley.edu/trac/wiki
http://www.boinc-wiki.info

The problem I'm having is that these are both wikis. So when downloading with e.g.:

wget -r -k -np -nv -R jpg,jpeg,gif,png,tif http://www.boinc-wiki.info/

I do get a lot of files because it also follows links like ...?action=edit ...?action=diff&version=...

Does somebody know a way to get around this?

I just want the current pages, without images, and without diffs etc.

P.S.:

wget -r -k -np -nv -l 1 -R jpg,jpeg,png,gif,tif,pdf,ppt http://boinc.berkeley.edu/trac/wiki/TitleIndex

This worked for berkeley but boinc-wiki.info is still giving me trouble :/

P.P.S:

I got what appears to be the most relevant pages with:

wget -r -k -nv  -l 2 -R jpg,jpeg,png,gif,tif,pdf,ppt http://www.boinc-wiki.info
share|improve this question
    
No need to cross post between superuser and serverfault superuser.com/questions/158318/… –  Bryan Jun 29 '10 at 22:07
    
Where should I have posted it? –  Tie-fighter Jun 29 '10 at 22:25
add comment

6 Answers

The documentation for wget says:

Note, too, that query strings (strings at the end of a URL beginning with a question mark (‘?’) are not included as part of the filename for accept/reject rules, even though these will actually contribute to the name chosen for the local file. It is expected that a future version of Wget will provide an option to allow matching against query strings.

It looks like this functionality has been on the table for awhile and nothing has been done with it.

I haven't used it, but httrack looks like it has a more robust filtering feature set than wget and may be a better fit for what you're looking for (read about filters here http://www.httrack.com/html/fcguide.html).

share|improve this answer
    
+1 for pointing me to httrack. It looks better then wget, and wget is looking stagnant. –  Stefan Lasiewski Jun 29 '10 at 22:04
    
I've tried Winhttrack but it behaves funny. It downloads files and traverses directories it should not :/ –  Tie-fighter Jun 29 '10 at 22:15
    
Maybe one day wget will be fixed. For now httrack and pavuk both look good. –  joeytwiddle Jan 12 '12 at 22:18
add comment

The new version of wget (v.1.14) solves all these problems.

You have to use the new option --reject-regex=.... to handle query strings.

Note that I couldn't find the new manual that includes these new options, so you have to use the help command wget --help > help.txt

share|improve this answer
add comment

Pavuk should be able to do it:

http://pavuk.sourceforge.net/man.html#sect39

Mediawiki example:

[...]

-skip_url_pattern ’oldid=, action=edit, action=history, diff=, limit=, [/=]User:, *[/=]User_talk:*, [^p]/Special:, =Special:[^R], .php/Special:[^LUA][^onl][^nul], MediaWiki:, Search:, Help:

[...]

share|improve this answer
add comment

It looks like you are trying to avoid download special pages of MediaWiki. I solved this problem once avoiding the index.php page:

wget  -R '*index.php*'  -r ... <wiki link>

However, the wiki used the URLS as seen in Wikipedia (http://<wiki>/en/Theme) and not the pattern I have seen in other places (http://<wiki>/index.php?title=Theme). Since the link you gave uses URLs in the Wikipedia pattern, I think this solution can work for you too, though.

share|improve this answer
add comment
wget --reject-regex '(.*)\?(.*)' http://example.com

(--reject-type posix by default). Works only for recent (>=1.14) versions of wget though, according to other comments.

Beware that it seems you can use --reject-regex only once per wget call. That is, you have to use | in a single regex if you want to select on several regex :

wget --reject-regex 'expr1|expr2|…' http://example.com
share|improve this answer
add comment

‘-R rejlist --reject rejlist’ Specify comma-separated lists of file name suffixes or patterns to accept or reject (see Types of Files). Note that if any of the wildcard characters, ‘*’, ‘?’, ‘[’ or ‘]’, appear in an element of acclist or rejlist, it will be treated as a pattern, rather than a suffix.

Patterns are probably what you want. I am not sure how sophisticated the patterns are but you can either try to accept only certain files or block:

wget -r -k -np -nv -R jpg,jpeg,gif,png,tif,*\? http://www.boinc-wiki.info/

Accept:

wget -r -k -np -nv -R jpg,jpeg,gif,png,tif -A [a-zA-Z.] http://www.boinc-wiki.info/

Edit: nvm in light of the other post.

share|improve this answer
    
This works on query strings? Every version of wget I've used only applies reject list patterns to the file portion of the URL. I'll give it a shot and see. –  Evan Anderson Jun 29 '10 at 21:45
    
I haven't tested it. I just looked up the documentation. I did find it uses shell convention, but your experience would speak more than mine in regard to the working function of the matching. –  Joshua Enfield Jun 29 '10 at 21:47
    
Escaping the "?" doesn't seem to get wget to do what the OP would like on my CentOS 5.3 box running wget 1.11.4. –  Evan Anderson Jun 29 '10 at 21:50
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.