up vote 7 down vote favorite
1
share [g+] share [fb]

I've started a relatively new website, and I submitted it to google and everything. I use google's webmaster tools as well. I'm wondering how to figure out the frequency of google's spider accessing my website. I always hear people talking in forums about "I'm getting 250 pages crawled an hour" and so on... Is there software to use? A javascript method similar to google analytics? Or is there something to analyze my server's logs? Thanks

link|improve this question
feedback

migrated from stackoverflow.com Jan 23 '10 at 23:33

This question came from our site for professional and enthusiast programmers.

8 Answers

up vote 3 down vote

Just check the webserver logs if the visitor's User-Agent request header matches/contains Googlebot. There are lot of webserver log analyzer tools, either free or payware. Most of them are also able to categorize bots. From them all I've had the best experience with Google Analytics.

link|improve this answer
Ah, thanks. Can you tell me what the log file will probably be named? I'm using a 1and1 server and I'm fairly new to the web hosting game – RobHardgood Jan 23 '10 at 23:17
You're using 3rd party hosting? Well, contact them for more details or poke a bit round in the hosting's control panel. – BalusC Jan 23 '10 at 23:18
By the way, with Google Analytics you don't need to know about webserver logs at all. It runs completely independently. – BalusC Jan 23 '10 at 23:33
feedback
up vote 2 down vote

Signup for Google Webmaster Central, aka Webmasters Tools, verify your site, and wait for the stats to appear.

link|improve this answer
feedback
up vote 1 down vote

It's fairly unlikely Google Bot will run any Javascript on your site that will track it arriving, so the best hope is to look at the site logs themselves - either in webalizer or the hundreds of other log file parsers, or just open them up in a texteditor and run a regex to extract the Google Bot user agent.

You should see then how many hits it makes in a space of time. If it's too much you might want to look at using an XML sitemap

link|improve this answer
Yeah, I didn't think it would run any javascript. Thanks for the suggestions – RobHardgood Jan 23 '10 at 23:19
feedback
up vote 1 down vote

AWstats is a good, free utility to use for log file analysis. It'll give you a lot of interesting statistics in addition to the Google crawl info.

link|improve this answer
Ah, I just found this piece of software... It looks like it'll do the trick. Thanks! – RobHardgood Jan 23 '10 at 23:21
You're welcome. If this is the solution you're going with, you should accept the answer :) – Mahmoud Al-Qudsi Jan 23 '10 at 23:28
feedback
up vote 0 down vote

I would recommend Google analytics, it covers all visits to your site and you can work out whether a visitor was a bot or not. It is also excellent for SEO on your site helping to increase visitors.

link|improve this answer
Google Analytics doesn't include bots because it uses javascript, which bots generally don't execute. – hmallett Jan 24 '10 at 0:34
feedback
up vote 0 down vote

AWstats is a good, free utility to use for log file analysis. It'll give you a lot of interesting statistics in addition to the Google crawl info.

Awstats is a great tool indeed

link|improve this answer
feedback
up vote 0 down vote

Check this article which tells how to send mail when Googlebot crawls a webpage of your website

link|improve this answer
feedback
up vote 0 down vote

You can use googles webmaster tools, It will tell you how often the bot has crawled your site...

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard