Questions tagged [web-crawler]
A web-crawler (also known as a web-spider) traverses the webpages of the internet by following the links of urls contained within each webpage. There is usually an initial seed of URLs from which the crawler is given to initialize its crawl.
4
questions
5
votes
2
answers
4k
views
Strange request in access.log, how to block?
I am using nginx on my own server, and I noticed a few days ago some strange request in my access.log :
77.50.217.37 - - [19/Aug/2011:17:50:50 +0200] "GET http://images.google.com/ HTTP/1.1" 200 151 "...
- 53
7
votes
4
answers
5k
views
How do I rate limit google's crawl of my IP block?
I have several sites in a /24 network that all get crawled by google on a pretty regular basis. Normally this is fine. However, when google starts crawling all the sites at the same time, the small ...
- 1,042
2
votes
2
answers
4k
views
Apache crashing with memory/cpu overload when google crawler visits site
I have a site with low traffic, less than 500 hits a day. It has a 6G of memory and it is way underutilized, on average 5% is in use. But as soon as googlebot establishes a connection to my webserver/...
- 9,434
1
vote
3
answers
10k
views
Blocking 'good' bots in nginx with multiple conditions for certain off-limits URL's where humans can go
After 2 days of searching/trying/failing I decided to post this here, I haven't found any example of someone doing the same nor what I tried seems to be working OK. I'm trying to send a 403 to bots ...
- 221