Trying to block Google bot and others from accessing some of my sites. Thing is I have one box that has a ton of virtual host files that do nothing more than do a proxy pass to other servers. I would ...
Every day my access-log looks kind of this: 22.214.171.124 - - [21/Oct/2013:14:37:00 +0200] "GET /robots.txt HTTP/1.1" 200 112 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; ...
I have an angular app served through nginx. For googlebot I want to serve a different static html file so that it can index properly, is the following nginx config correct? (I don't want to complicate ...
I am getting repeated requests, logged in our Apache error logs, from 126.96.36.199 (which appears to be a googlebot), for a URL that is apparently good; the request then gets 404'd and our custom 404 ...