I have a link to an http page that has a structure like this:

Parent Directory -
[DIR] _OLD/ 01-Feb-2012 06:05 -
[DIR] _Jan/ 01-Feb-2012 06:05 -
[DIR] _Dec/ 01-Jan-2012 06:05 -
...... [DIR] _Apr/ 01-May-2011 06:05 -
[DIR] _Mar/ 01-Apr-2011 06:05 -
[DIR] _Feb/ 01-Mar-2011 06:05 -
[DIR] WEB-INF/ 21-Aug-2009 13:44 -
[ ] nohup_XXX_XXX21.out 14-Feb-2012 09:05 1.6M
[ ] XXX_XXX21.log 14-Feb-2012 09:04 64K
[ ] XXX_XXX21_access.log 14-Feb-2012 08:31 8.0K
[ ] XXX_XXX21_access.log00013 14-Feb-2012 00:01 585K

I would like to downlload only the files present in the root directory...the xxxx files.

I have a solution using

curl -U Mozilla http://yourpage.com/bla.html > page
grep -o http://[^[:space:]]*.*log* page > links
wget -i link

but i wonder is not possible to do that only using wget ?


All files from root directory matching pattern *.log*:

wget --user-agent=Mozilla --no-directories --accept='*.log*' -r -l 1 http://yourpage.com/bla.html
  • --user-agent=Mozilla set User-Agent header
  • --no-directories save all files in current directory
  • --accept='*.log' accepted extensions (pattern)
  • -r recursive
  • -l 1 one level of recursion

You avoid grepping out html links (could be error prone) at a cost of few more requests to server.

  • tried but does not work yet – Cris Feb 14 '12 at 12:36
  • @Cris Could you post command output somewhere? It works for me on similar URL. – kupson Feb 14 '12 at 12:48
  • @Cris I added --user-agent wget option to mimic your curl command more. – kupson Feb 14 '12 at 12:58
  • pastebin.com/0U5pWYfz Thanks – Cris Feb 14 '12 at 14:01
  • @cris Can you confirm that there are links <a href="http://..."> to XXX_XXX21.log files in first index.html? – kupson Feb 14 '12 at 14:13

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.