Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I came across an open source crawler that recently hit my site. I was wondering, 1. How do you get a list of sites to crawl? 2. Can you get a list of sites to crawl in your city? 3. If you have all this information, where is this readily available from and what APIs are there to access this information?

Thanks, Walter

share|improve this question
Belongs on StackOverflow. But to be helpful, you need to seed your crawler with links and then let it explore. If you wanted to explore sites from your city, you'd need to start with a say, "Boston" portal type site and go from there. There's no real master list or API for it, it's clever programming and focus upon target sites. – Xorlev Oct 25 '09 at 21:03
Ok, I should ask it there instead. I thought all companies have to register and that information was a public record. I already have a simple crawler, need to give it some starting information though. – Walter White Oct 26 '09 at 3:01

To start crawling, normally you hand the crawler an address, and let it run.

If you want to look only for "local" results, you may need to get clever with your spawning of the crawling process, and seed it with known URLs of local companies, which you might need to find the old-fashioned way: eg using a phonebook :)

share|improve this answer
Ok, I was hoping there was some all knowing source, but then again if there why, what would I need this for? Yeah, the crawler works fine for my internal site finding everything in there, but the problem is just giving it the initial address(es) to start from. – Walter White Oct 26 '09 at 12:24
up vote 0 down vote accepted

I was hoping for more ideas as to what other information is out there for me to reference and then do intelligent things with. I have a rough idea of what is, of course, I can always use Hacking Exposed to find other sources that I originally hadn't thought of.

share|improve this answer

To find local websites you can use those websites that do use Geo Coordinates as your starting points. And then while indexing sites without geo coordinates you have to check them for referencing certain address relevant strings (as in the area zip/postal code and the citys name).

You can use this geourl websites index to automate the process by using certain long/lat values as a parameter like i did for fresno here:

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.