Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a link to an http page that has a structure like this:

Parent Directory -
[DIR] _OLD/ 01-Feb-2012 06:05 -
[DIR] _Jan/ 01-Feb-2012 06:05 -
[DIR] _Dec/ 01-Jan-2012 06:05 -
...... [DIR] _Apr/ 01-May-2011 06:05 -
[DIR] _Mar/ 01-Apr-2011 06:05 -
[DIR] _Feb/ 01-Mar-2011 06:05 -
[DIR] WEB-INF/ 21-Aug-2009 13:44 -
[ ] nohup_XXX_XXX21.out 14-Feb-2012 09:05 1.6M
[ ] XXX_XXX21.log 14-Feb-2012 09:04 64K
[ ] XXX_XXX21_access.log 14-Feb-2012 08:31 8.0K
[ ] XXX_XXX21_access.log00013 14-Feb-2012 00:01 585K

I would like to downlload only the files present in the root directory...the xxxx files.

I have a solution using

curl -U Mozilla http://yourpage.com/bla.html > page
grep -o http://[^[:space:]]*.*log* page > links
wget -i link

but i wonder is not possible to do that only using wget ?

share|improve this question
add comment

1 Answer

up vote 4 down vote accepted

All files from root directory matching pattern *.log*:

wget --user-agent=Mozilla --no-directories --accept='*.log*' -r -l 1 http://yourpage.com/bla.html
  • --user-agent=Mozilla set User-Agent header
  • --no-directories save all files in current directory
  • --accept='*.log' accepted extensions (pattern)
  • -r recursive
  • -l 1 one level of recursion

You avoid grepping out html links (could be error prone) at a cost of few more requests to server.

share|improve this answer
    
tried but does not work yet –  Cris Feb 14 '12 at 12:36
    
@Cris Could you post command output somewhere? It works for me on similar URL. –  kupson Feb 14 '12 at 12:48
    
@Cris I added --user-agent wget option to mimic your curl command more. –  kupson Feb 14 '12 at 12:58
    
pastebin.com/0U5pWYfz Thanks –  Cris Feb 14 '12 at 14:01
    
@cris Can you confirm that there are links <a href="http://..."> to XXX_XXX21.log files in first index.html? –  kupson Feb 14 '12 at 14:13
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.