2

I am trying to mirror most of a ftp sever using wget.

This particular sever keeps old copies of the data files in a folder called 'backup' inside every subfolder. e.g. '/MaffiaOffShore/backup' '/VeryVeryDodgy/backup', which I don't want but my attempts at using --reject 'backup' and using --exclude-directories 'backup' and various other combinations, all have the same result, the whole sever is mirrored, including the backup folder.

Is this a bug or am I missing something?

3 Answers 3

0

The exclude directories option is:

--exclude-directories=List

not --exclude-directory as you have included in your question

from the wget man page

-X list
--exclude-directories=list
   Specify a comma-separated list of directories you wish to exclude
   from download (@pxref{Directory-Based Limits} for more details.)
   Elements of list may contain wildcards.
4
  • typo, but the correct version doesn't work either :-(
    – WgetMonkey
    Jun 22, 2009 at 9:12
  • I've just updated my answer '*/backup', which was initially not 100% correct
    – kubanczyk
    Jun 22, 2009 at 9:18
  • That works now, thanks. Not the most 'user-friendly' way of writing the software...
    – WgetMonkey
    Jun 22, 2009 at 9:32
  • Be aware that the folders need to be absolute paths! So to block host.org/img/thumbs/ you have to write --exclude-directories=/img/thumbs/.
    – user136036
    Apr 2, 2021 at 22:01
0

If you want to exclude /MaffiaOffShore/backup from /MaffiaOffShore/, try

-x/MaffiaOffShore/backup

or,

--exclude-directories=/MaffiaOffShore/backup 

explicitly.


So, from the base directory you exclude all un-required directories as,

-x/MaffiaOffShore/backup,/VeryVeryDodgy/backup
0

Yes, your attempts fail also on my wget. However, I've just verified that the following works nicely for all nested subdirectories:

wget -r ftp://user@xxx --exclude-directories '/backup,*/backup,*/*/backup,*/*/*/backup'

The wildcarded '*/backup' will not match /backup but it will match paths like /something/backup. The wildcarded '*/*/backup' will match '/something/else/backup', etc.

2
  • Just tried that, didn't work. One combination I tried gave me a backtrace and crashed! I'm using version 'GNU Wget 1.11.4', what were you using? Are you sure that backup folders existed on the sever you tested this on?
    – WgetMonkey
    Jun 22, 2009 at 9:24
  • Mine is wget 1.10.2 so it's older. I've created a test directory tree with backup folders all over it. I guess the problem might be that your tree contains a very large number of directories, so the /*/ expands to a big list in the memory?
    – kubanczyk
    Jun 22, 2009 at 9:54

You must log in to answer this question.