Trying to block Google bot and others from accessing some of my sites. Thing is I have one box that has a ton of virtual host files that do nothing more than do a proxy pass to other servers. I would ...
I just updated my robots.txt file on a new site; Google Webmaster Tools reports it read my robots.txt 2 days before my last update. my last robots.txt had a "disallow: all" raw. Is there any way I ...
Every day my access-log looks kind of this: 18.104.22.168 - - [21/Oct/2013:14:37:00 +0200] "GET /robots.txt HTTP/1.1" 200 112 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; ...
I have an angular app served through nginx. For googlebot I want to serve a different static html file so that it can index properly, is the following nginx config correct? (I don't want to complicate ...
I am getting repeated requests, logged in our Apache error logs, from 22.214.171.124 (which appears to be a googlebot), for a URL that is apparently good; the request then gets 404'd and our custom 404 ...