1

I have a link to an http page that has a structure like this:

Parent Directory -
[DIR] _OLD/ 01-Feb-2012 06:05 -
[DIR] _Jan/ 01-Feb-2012 06:05 -
[DIR] _Dec/ 01-Jan-2012 06:05 -
...... [DIR] _Apr/ 01-May-2011 06:05 -
[DIR] _Mar/ 01-Apr-2011 06:05 -
[DIR] _Feb/ 01-Mar-2011 06:05 -
[DIR] WEB-INF/ 21-Aug-2009 13:44 -
[ ] nohup_XXX_XXX21.out 14-Feb-2012 09:05 1.6M
[ ] XXX_XXX21.log 14-Feb-2012 09:04 64K
[ ] XXX_XXX21_access.log 14-Feb-2012 08:31 8.0K
[ ] XXX_XXX21_access.log00013 14-Feb-2012 00:01 585K

I would like to downlload only the files present in the root directory...the xxxx files.

I have a solution using

curl -U Mozilla http://yourpage.com/bla.html > page
grep -o http://[^[:space:]]*.*log* page > links
wget -i link

but i wonder is not possible to do that only using wget ?

1 Answer 1

11

All files from root directory matching pattern *.log*:

wget --user-agent=Mozilla --no-directories --accept='*.log*' -r -l 1 http://yourpage.com/bla.html
  • --user-agent=Mozilla set User-Agent header
  • --no-directories save all files in current directory
  • --accept='*.log' accepted extensions (pattern)
  • -r recursive
  • -l 1 one level of recursion

You avoid grepping out html links (could be error prone) at a cost of few more requests to server.

7
  • tried but does not work yet
    – Cris
    Feb 14, 2012 at 12:36
  • @Cris Could you post command output somewhere? It works for me on similar URL.
    – kupson
    Feb 14, 2012 at 12:48
  • @Cris I added --user-agent wget option to mimic your curl command more.
    – kupson
    Feb 14, 2012 at 12:58
  • pastebin.com/0U5pWYfz Thanks
    – Cris
    Feb 14, 2012 at 14:01
  • @cris Can you confirm that there are links <a href="http://..."> to XXX_XXX21.log files in first index.html?
    – kupson
    Feb 14, 2012 at 14:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .