I came across an open source crawler that recently hit my site. I was wondering, 1. How do you get a list of sites to crawl? 2. Can you get a list of sites to crawl in your city? 3. If you have all this information, where is this readily available from and what APIs are there to access this information?
Thanks, Walter