0

I need to analyze the search engine crawling going on in my site. Is there a good tool for this? I've tried AWStats and Sawmill. But both of those give me very limited insight into the crawling. I need to know information like how many unique/distinct webpages in a section of my site was crawled by a specific crawler within a time period.

Google analytics doesn't track crawling at all due to its javascript tracking mechanism.

1 Answer 1

0

Clarity is a front end interface similar to Splunk.

Logstash should be able to help you with what you need.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .