For the last few days, I've been having a lot of requests to unexisting pages on my server. The worrying part about it is that, when I look at the 404 error log I've built for my site, these requests ...
I've started tracking user-agent strings on a website at the start of each session. Looking at the data for this month so far I'm seeing on search engine bot that keeps coming up a lot.. Mozilla/5.0 ...
I need to analyze the search engine crawling going on in my site. Is there a good tool for this? I've tried AWStats and Sawmill. But both of those give me very limited insight into the crawling. I ...