The tag has no wiki summary.

learn more… | top users | synonyms

0
votes
1answer
54 views

apache rewrite syntax

Trying to block Google bot and others from accessing some of my sites. Thing is I have one box that has a ton of virtual host files that do nothing more than do a proxy pass to other servers. I would ...
2
votes
0answers
147 views

Google-Bot fell in love with my 404-page

Every day my access-log looks kind of this: 66.249.78.140 - - [21/Oct/2013:14:37:00 +0200] "GET /robots.txt HTTP/1.1" 200 112 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; ...
1
vote
0answers
77 views

nginx serve different html file for googlebot

I have an angular app served through nginx. For googlebot I want to serve a different static html file so that it can index properly, is the following nginx config correct? (I don't want to complicate ...
0
votes
0answers
73 views

Googlebot causing 404 for file that exists

I am getting repeated requests, logged in our Apache error logs, from 66.249.76.18 (which appears to be a googlebot), for a URL that is apparently good; the request then gets 404'd and our custom 404 ...