Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

i'm crawling a forum and I keep stumbling across certain threads that have been going on for ten years.

i can certainly exclude these using wget option:

-X /t/41866,/t/314849,/t/335041,/t/356321,/t/491462,/t/493609,/t/493655,/t/493667,/t/493668,/t/493676,/t/493678

and I can also exclude them by inserting the string in the wgetrc file

but what i'd like to do is just call a file that contains the string, like you can do with the -i option where you call a file that pulls in the URLs of interest

so instead of (from the GNU wget 1.11.4 manual)

exclude directories = string Specify a comma-separated list of directories you wish to exclude from download— the same as ‘-X string’

so i'd like the string to actually pull in the contents of a file. is there a way to do this?

share|improve this question
    
Welcome to Server Fault! Your question is off topic for Serverfault because it doesn't appear to relate to servers/networking or desktop infrastructure in a professional environment. It may be on topic for Superuser but please search their site for similar questions that may already have the answer you're looking for. –  Dennis Kaarsemaker Feb 17 '13 at 20:28

3 Answers 3

You can use the -I list or --include-directories=list option:

   -I list
   --include-directories=list
       Specify a comma-separated list of directories you wish to follow when downloading.  Elements
       of list may contain wildcards.
share|improve this answer
wget -X `perl -MFile::Slurp -e '@lines=read_file("./FILE.txt"); chop @lines; print join ",", @lines'`

(you may need to install File::Slurp Perl module).

share|improve this answer

You could always just use the shell

wget -X `head -n1 exclude_file` blah

the head -n1 is incase there is a trailing newline in the file.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.