Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [googlebot]

The tag has no usage guidance.

-2
votes
1answer
95 views

Googelbot finds my original URI, although I have a working rewrite directive [closed]

I have : RewriteRule ^Article/([^/]*)$ /article.php?newsid=$1 [L] Which means that the URL must be //example.com/Article/855563 but Google crawls //example.com/article.php?newsid=855563. Is there ...
0
votes
1answer
2k views

Block googlebot on a specific page using nginx

We're currently being crawled at a greater rate than we can handle. I can't seem to get nginx blocking the googlebot server { location /ajax/sse.php { if ($http_user_agent ~* "Mozilla/5....
1
vote
0answers
287 views

Enabling TLS/SSL with SNI on a subset of websites, without losing SEO ranking on the non-TLS sites

We run a number of LAMP servers on AWS with a few dozen websites on them, that customers pay us to design, build and host. They're Ubuntu 14.04 servers with Varnish, Apache and PHP. Currently, if a ...
1
vote
1answer
857 views

How to prevent Google Favicon bot to call to my site?

I have some backend url that I use for myself in google chrome only. It's not open public. However for some reason, this bot "Google Favicon" ip located at Google call this URL which I do not want. My ...
2
votes
1answer
266 views

What's with random-character queries coming from googlebot, e.g., vvytnoxvontwusz.html?

One of my sites has been getting queries from googlebot, on the order of: example-log:66.249.79.216 - - [06/Apr/2016:15:36:56 -0700] "GET /vvytnoxvontwusz.html HTTP/1.1" 404 15136 "-" "Mozilla/5.0 (...
1
vote
1answer
121 views

Allow Google To Bypass Firewall Nginx

So I am looking for a system which essentially returns a 401 for every visitor that doesn't have a certain cookie. I would like to make it so if the visitor/requester is google then it does not return ...
0
votes
1answer
193 views

Moved website to new server - updated DNS - web crawlers still hitting old site by IP

About ten days ago I moved a site - mostly a Joomla discussion board - to a new server at a different IP address. During a brief scheduled downtime I replicated the content over and completed DNS ...
1
vote
2answers
1k views

block fake google bots [duplicate]

How could I block DDOS attacks with fake Google bots? I found 2 solutions on the net. But both seems to block also correct google bots. # Block fake google when it's not coming from their IP range's ...
0
votes
1answer
133 views

Google bot cannot read my web site

I am getting from time to time a message from Google bot that it cannot access my web site. Over the last 24 hours, Googlebot encountered 1 errors while attempting to retrieve DNS information ...
4
votes
2answers
24k views

AH01797: client denied by server configuration: /usr/share/doc

Since quite a while (over a month now) I see lines like the following in the apache logs: 180.76.15.138 - - [24/Jun/2015:16:13:34 -0400] "GET /manual/de/mod/module-dict.html HTTP/1.1" 403 396 "-" "...
1
vote
2answers
88 views

Googlebot requesting pages of 1 site on another site

Problem: Using Prerender.io to index/store pages of one site, I keep getting path requests that only exist on my old site Example: on Prerender I'll see that Googlebot requested http://www.new-site....
1
vote
0answers
483 views

Trouble filtering googlebot from apache access log

Though it seems like it should be pretty straightforward, I have been unable to configure apache so that googlebot's requests are not stored in the access log. I've tried the following lines: ...
0
votes
0answers
157 views

Apache duplicate every GET request made by Googlebot

System: Linux 3.10.47.core2.24 Apache: Most likely version 2.2 (can`t check that) Server API: Apache 2.0 Handler Apache API Version: 20051115 In logs requests looks like this: 94.*.*.* - - [26/Nov/...
-4
votes
1answer
179 views

How Can I Encourage Google to scan New robots.txt File?

I just updated my robots.txt file on a new site; Google Webmaster Tools reports it read my robots.txt 2 days before my last update. my last robots.txt had a "disallow: all" raw. Is there any way I ...
1
vote
1answer
809 views

Google-bot trips on a perfectly normal robots.txt, then on a nonexistent robots.txt [duplicate]

I have two domain names pointing to the same virtual server. One of them, http://ilarikaila.com, is a working brochure website I made for a friend. I used the other one, http://teemuleisti.com, to ...
2
votes
1answer
2k views

fail2ban ignoreip DNS host example?

I would like to add ".googlebot.com" to the ignore iplist for fail2ban since the ignoreip explanation mentions DNS host as an accepted input. Is this a proper format? # "ignoreip" can be an IP ...
3
votes
1answer
245 views

Why is googlebot requesting robots.txt from my SSH server?

I run ossec on my server and periodically I receive a warning like this: Received From: myserver->/var/log/auth.log Rule: 5701 fired (level 8) -> "Possible attack on the ssh server (or version ...
1
vote
1answer
265 views

Googlebot cant access my site webmaster tools reply Unreachable robots.txt

When I try to fetch my site as a googlebot in webmaster tools it return Unreachable robots.txt, after investigate I understood google bot can see my server: tcpdump | grep google It returns that ...
2
votes
1answer
830 views

Google-Bot fell in love with my 404-page

Every day my access-log looks kind of this: 66.249.78.140 - - [21/Oct/2013:14:37:00 +0200] "GET /robots.txt HTTP/1.1" 200 112 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot....
0
votes
0answers
1k views

nginx serve different html file for googlebot

I have an angular app served through nginx. For googlebot I want to serve a different static html file so that it can index properly, is the following nginx config correct? (I don't want to complicate ...
2
votes
1answer
529 views

Nginx Googlebot rewrite rules failing with 404

Our site is based on Angular which makes it almost completely JavaScript based, therefore we need to serve static HTML snapshots to the Googlebot in order for it to crawl us. At the moment, we have ...
0
votes
1answer
135 views

apache rewrite syntax

Trying to block Google bot and others from accessing some of my sites. Thing is I have one box that has a ton of virtual host files that do nothing more than do a proxy pass to other servers. I would ...
0
votes
1answer
53 views

Googlebot incrementing page id

So here is an example of a hit I'm getting from the googlebot: 66.249.73.171 - - [19/Feb/2013:16:12:39 -0500] "GET /eghm-blah.php?pid=2855 HTTP/1.1" 200 1684 "-" "Mozilla/5.0 (compatible; Googlebot/2....
1
vote
1answer
274 views

Googlebot repeatedly looks for files that aren't on my server

I'm hosting a site for a volunteer organization. I've moved the site to WordPress, but it wasn't always that way. I suspect at one point it was hacked badly. My Apache error log file has grown to ...
2
votes
1answer
382 views

Apache : with Googlebot connections, a single process takes all server memory

Following https://serverfault.com/questions/418735/unbelievable-issue-a-single-apache-process-takes-4-gb-of-memory I post as a new question because I was able to identify the fact that it happens when ...
0
votes
1answer
691 views

How to fix googlebot Server Connectivity

I get 'Server Connectivity' error at google webmaster tool. I suspect it is because of iptables rules that I've set to counter some DDoS attacks, thugh I'm not sure which rules could be relevant. This ...