The crawl-rate tag has no wiki summary.
3
votes
4answers
2k views
How do I rate limit google's crawl of my IP block?
I have several sites in a /24 network that all get crawled by google on a pretty regular basis. Normally this is fine. However, when google starts crawling all the sites at the same time, the small ...
0
votes
1answer
501 views
Massive Crawling requests from Google Apps Engine useragent
I'm badly affected with 'Google AppEngine-Google' UserAgent.. receiving 5/6 requests per second on http server. This bot is crawling my site just like GoogleBot does.
Following is the sample of url ...
