I am using nginx on my own server, and I noticed a few days ago some strange request in my access.log : 18.104.22.168 - - [19/Aug/2011:17:50:50 +0200] "GET http://images.google.com/ HTTP/1.1" 200 151 ...
Blocking 'good' bots in nginx with multiple conditions for certain off-limits URL's where humans can go
After 2 days of searching/trying/failing I decided to post this here, I haven't found any example of someone doing the same nor what I tried seems to be working OK. I'm trying to send a 403 to bots ...
I have a site with low traffic, less than 500 hits a day. It has a 6G of memory and it is way underutilized, on average 5% is in use. But as soon as googlebot establishes a connection to my ...