Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'd like to scan all the url's on my website as well as get the files in them, but the thing is, there are too many for me to do this manually so how would I do this?

I'd like it formatted anyway as long as there is some type of order to it.


All in a file like a .txt

How would I do that?

share|improve this question
What about wget, find, ls? – Deer Hunter Dec 27 '12 at 8:21
up vote 2 down vote accepted

Try ls passing the -R switch. It lists subdirectories. Here is an example:

ls -R /path/to/whatever > folders.txt 
share|improve this answer
Sorry, its kinda late. But that was perfect. I was busy fixing everything. – user151015 Jan 23 '13 at 0:07

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.