-1

I came across an open source crawler that recently hit my site. I was wondering, 1. How do you get a list of sites to crawl? 2. Can you get a list of sites to crawl in your city? 3. If you have all this information, where is this readily available from and what APIs are there to access this information?

Thanks, Walter

2
  • Belongs on StackOverflow. But to be helpful, you need to seed your crawler with links and then let it explore. If you wanted to explore sites from your city, you'd need to start with a say, "Boston" portal type site and go from there. There's no real master list or API for it, it's clever programming and focus upon target sites.
    – Xorlev
    Oct 25, 2009 at 21:03
  • Ok, I should ask it there instead. I thought all companies have to register and that information was a public record. I already have a simple crawler, need to give it some starting information though.
    – Walter White
    Oct 26, 2009 at 3:01

3 Answers 3

0

To start crawling, normally you hand the crawler an address, and let it run.

If you want to look only for "local" results, you may need to get clever with your spawning of the crawling process, and seed it with known URLs of local companies, which you might need to find the old-fashioned way: eg using a phonebook :)

1
  • Ok, I was hoping there was some all knowing source, but then again if there why, what would I need this for? Yeah, the crawler works fine for my internal site finding everything in there, but the problem is just giving it the initial address(es) to start from.
    – Walter White
    Oct 26, 2009 at 12:24
0

I was hoping for more ideas as to what other information is out there for me to reference and then do intelligent things with. I have a rough idea of what is, of course, I can always use Hacking Exposed to find other sources that I originally hadn't thought of.

0

To find local websites you can use those websites that do use Geo Coordinates as your starting points. And then while indexing sites without geo coordinates you have to check them for referencing certain address relevant strings (as in the area zip/postal code and the citys name).

You can use this geourl websites index to automate the process by using certain long/lat values as a parameter like i did for fresno here: http://geourl.org/near?lat=36.78&long=-119.79

You must log in to answer this question.