The tag has no wiki summary.

learn more… | top users | synonyms

0
votes
1answer
66 views

apache rewrite syntax

Trying to block Google bot and others from accessing some of my sites. Thing is I have one box that has a ton of virtual host files that do nothing more than do a proxy pass to other servers. I would ...
-4
votes
1answer
51 views

How Can I Encourage Google to scan New robots.txt File?

I just updated my robots.txt file on a new site; Google Webmaster Tools reports it read my robots.txt 2 days before my last update. my last robots.txt had a "disallow: all" raw. Is there any way I ...
2
votes
0answers
234 views

Google-Bot fell in love with my 404-page

Every day my access-log looks kind of this: 66.249.78.140 - - [21/Oct/2013:14:37:00 +0200] "GET /robots.txt HTTP/1.1" 200 112 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; ...
1
vote
0answers
160 views

nginx serve different html file for googlebot

I have an angular app served through nginx. For googlebot I want to serve a different static html file so that it can index properly, is the following nginx config correct? (I don't want to complicate ...
0
votes
0answers
28 views

Apache duplicate every GET request made by Googlebot

System: Linux 3.10.47.core2.24 Apache: Most likely version 2.2 (can`t check that) Server API: Apache 2.0 Handler Apache API Version: 20051115 In logs requests looks like this: 94.*.*.* - - ...
0
votes
0answers
123 views

Googlebot causing 404 for file that exists

I am getting repeated requests, logged in our Apache error logs, from 66.249.76.18 (which appears to be a googlebot), for a URL that is apparently good; the request then gets 404'd and our custom 404 ...