0
votes
1answer
97 views

Web log file analysis software to measure search crawlers

I need to analyze the search engine crawling going on in my site. Is there a good tool for this? I've tried AWStats and Sawmill. But both of those give me very limited insight into the crawling. I ...
0
votes
2answers
121 views

Are there any regularly updated IP databases of Bots?

I am looking for a database of IP Addresses of known bots, so I can identify them in web site traffic. I have useragent data, but there seem to be plenty that don't give useragents to identify ...
2
votes
1answer
227 views

Are there any regularly updated Bot/Spider/Crawler Databases?

I am looking for a database that is regularly updated of different bots, spiders, and crawlers. I want to be able to identify them in the log files from IIS.