I need to analyze the search engine crawling going on in my site. Is there a good tool for this? I've tried AWStats and Sawmill. But both of those give me very limited insight into the crawling. I ...
I am looking for a database of IP Addresses of known bots, so I can identify them in web site traffic. I have useragent data, but there seem to be plenty that don't give useragents to identify ...
I am looking for a database that is regularly updated of different bots, spiders, and crawlers. I want to be able to identify them in the log files from IIS.