Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I've started a relatively new website, and I submitted it to google and everything. I use google's webmaster tools as well. I'm wondering how to figure out the frequency of google's spider accessing my website. I always hear people talking in forums about "I'm getting 250 pages crawled an hour" and so on... Is there software to use? A javascript method similar to google analytics? Or is there something to analyze my server's logs? Thanks

share|improve this question

migrated from stackoverflow.com Jan 23 '10 at 23:33

This question came from our site for professional and enthusiast programmers.

8 Answers 8

Just check the webserver logs if the visitor's User-Agent request header matches/contains Googlebot. There are lot of webserver log analyzer tools, either free or payware. Most of them are also able to categorize bots. From them all I've had the best experience with Google Analytics.

share|improve this answer
    
Ah, thanks. Can you tell me what the log file will probably be named? I'm using a 1and1 server and I'm fairly new to the web hosting game –  RobHardgood Jan 23 '10 at 23:17
    
You're using 3rd party hosting? Well, contact them for more details or poke a bit round in the hosting's control panel. –  BalusC Jan 23 '10 at 23:18
    
By the way, with Google Analytics you don't need to know about webserver logs at all. It runs completely independently. –  BalusC Jan 23 '10 at 23:33

Signup for Google Webmaster Central, aka Webmasters Tools, verify your site, and wait for the stats to appear.

share|improve this answer

It's fairly unlikely Google Bot will run any Javascript on your site that will track it arriving, so the best hope is to look at the site logs themselves - either in webalizer or the hundreds of other log file parsers, or just open them up in a texteditor and run a regex to extract the Google Bot user agent.

You should see then how many hits it makes in a space of time. If it's too much you might want to look at using an XML sitemap

share|improve this answer
    
Yeah, I didn't think it would run any javascript. Thanks for the suggestions –  RobHardgood Jan 23 '10 at 23:19

AWstats is a good, free utility to use for log file analysis. It'll give you a lot of interesting statistics in addition to the Google crawl info.

share|improve this answer
    
Ah, I just found this piece of software... It looks like it'll do the trick. Thanks! –  RobHardgood Jan 23 '10 at 23:21
    
You're welcome. If this is the solution you're going with, you should accept the answer :) –  Mahmoud Al-Qudsi Jan 23 '10 at 23:28

I would recommend Google analytics, it covers all visits to your site and you can work out whether a visitor was a bot or not. It is also excellent for SEO on your site helping to increase visitors.

share|improve this answer
1  
Google Analytics doesn't include bots because it uses javascript, which bots generally don't execute. –  hmallett Jan 24 '10 at 0:34

AWstats is a good, free utility to use for log file analysis. It'll give you a lot of interesting statistics in addition to the Google crawl info.

Awstats is a great tool indeed

share|improve this answer

Check this article which tells how to send mail when Googlebot crawls a webpage of your website

share|improve this answer

You can use googles webmaster tools, It will tell you how often the bot has crawled your site...

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.