The tag has no wiki summary.

learn more… | top users | synonyms

0
votes
1answer
78 views

apache rewrite syntax

Trying to block Google bot and others from accessing some of my sites. Thing is I have one box that has a ton of virtual host files that do nothing more than do a proxy pass to other servers. I would ...
-4
votes
1answer
100 views

How Can I Encourage Google to scan New robots.txt File?

I just updated my robots.txt file on a new site; Google Webmaster Tools reports it read my robots.txt 2 days before my last update. my last robots.txt had a "disallow: all" raw. Is there any way I ...
2
votes
0answers
339 views

Google-Bot fell in love with my 404-page

Every day my access-log looks kind of this: 66.249.78.140 - - [21/Oct/2013:14:37:00 +0200] "GET /robots.txt HTTP/1.1" 200 112 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; ...
1
vote
0answers
32 views

Googlebot requesting pages of 1 site on another site

Problem: Using Prerender.io to index/store pages of one site, I keep getting path requests that only exist on my old site Example: on Prerender I'll see that Googlebot requested ...
1
vote
0answers
286 views

nginx serve different html file for googlebot

I have an angular app served through nginx. For googlebot I want to serve a different static html file so that it can index properly, is the following nginx config correct? (I don't want to complicate ...
0
votes
0answers
61 views

Trouble filtering googlebot from apache access log

Though it seems like it should be pretty straightforward, I have been unable to configure apache so that googlebot's requests are not stored in the access log. I've tried the following lines: ...
0
votes
0answers
75 views

Apache duplicate every GET request made by Googlebot

System: Linux 3.10.47.core2.24 Apache: Most likely version 2.2 (can`t check that) Server API: Apache 2.0 Handler Apache API Version: 20051115 In logs requests looks like this: 94.*.*.* - - ...