A web-crawler (also known as a web-spider) traverses the webpages of the internet by following the links of urls contained within each webpage. There is usually an initial seed of URLs from which the crawler is given to initialize its crawl.

learn more… | top users | synonyms

4
votes
2answers
1k views

Strange request in access.log, how to block?

I am using nginx on my own server, and I noticed a few days ago some strange request in my access.log : 77.50.217.37 - - [19/Aug/2011:17:50:50 +0200] "GET http://images.google.com/ HTTP/1.1" 200 151 "...
7
votes
4answers
3k views

How do I rate limit google's crawl of my IP block?

I have several sites in a /24 network that all get crawled by google on a pretty regular basis. Normally this is fine. However, when google starts crawling all the sites at the same time, the small ...
1
vote
3answers
6k views

Blocking 'good' bots in nginx with multiple conditions for certain off-limits URL's where humans can go

After 2 days of searching/trying/failing I decided to post this here, I haven't found any example of someone doing the same nor what I tried seems to be working OK. I'm trying to send a 403 to bots ...
2
votes
2answers
3k views

Apache crashing with memory/cpu overload when google crawler visits site

I have a site with low traffic, less than 500 hits a day. It has a 6G of memory and it is way underutilized, on average 5% is in use. But as soon as googlebot establishes a connection to my webserver/...