Questions tagged [googlebot]
The googlebot tag has no usage guidance.
32
questions
0
votes
0
answers
58
views
Malicious Requests routed through 'Feedfetcher-Google' and Google Proxy IP
We're struggling with a unique situation where malicious/unauthorized requests are being made to our site via 'Google Proxy' IP addresses.
Someone is using Google servers to 'proxy' our website and ...
- 1
0
votes
0
answers
91
views
Slow server performance due to Google Bot and Ahrefs Bot
I have a VPS server with 64 CPU cores and 128 GB RAM running on Debian. I have noticed that Google Bot from IP address 66.249.66.0/24 and Ahrefs Bot from IP addresses 54.36.148.0/24 and 54.36.149.0/24 ...
1
vote
1
answer
202
views
High CPU load caused by bot traffic
Google bot crawl rate is every 2 seconds and it creates about 1.0-1.5 CPU load (average of 1 min) on a KVM host and a VM(web server) until the bot stops around 4AM.
If you see the graph, there is not ...
- 13
2
votes
1
answer
119
views
How can a nameserver block Google bot?
Background: Our domain page.et is not accessible by Google's mobile-friendly checking tool and search console. The same seems to be true for all other .et domains I tested.
The reason is not the ...
- 696
0
votes
2
answers
322
views
Will I be blocking the IP of some google related service?
In my sites I have created a script that sends me an email every time a new ip claiming to be google visits the site.
When I see the email I go to check (for example on whois.com) if the ip that ...
- 67
0
votes
0
answers
297
views
WAF(modsecurity) / Plesk IP Banned, is it Googlebot? Is it a false positive? Is it a malicious IP?
I was alerted by my Plesk server that an IP Address had been banned. Normally I don't check banned IPs, but this one happened to coincide with our site going down for 1 minute at the same time.
Banned ...
- 141
-2
votes
1
answer
98
views
Googelbot finds my original URI, although I have a working rewrite directive [closed]
I have :
RewriteRule ^Article/([^/]*)$ /article.php?newsid=$1 [L]
Which means that the URL must be //example.com/Article/855563
but Google crawls //example.com/article.php?newsid=855563.
Is there ...
1
vote
1
answer
7k
views
Block googlebot on a specific page using nginx
We're currently being crawled at a greater rate than we can handle.
I can't seem to get nginx blocking the googlebot
server {
location /ajax/sse.php {
if ($http_user_agent ~* "Mozilla/5....
- 291
1
vote
0
answers
393
views
Enabling TLS/SSL with SNI on a subset of websites, without losing SEO ranking on the non-TLS sites
We run a number of LAMP servers on AWS with a few dozen websites on them, that customers pay us to design, build and host. They're Ubuntu 14.04 servers with Varnish, Apache and PHP.
Currently, if a ...
- 7,748
1
vote
1
answer
1k
views
How to prevent Google Favicon bot to call to my site?
I have some backend url that I use for myself in google chrome only. It's not open public. However for some reason, this bot "Google Favicon" ip located at Google call this URL which I do not want. My ...
2
votes
1
answer
497
views
What's with random-character queries coming from googlebot, e.g., vvytnoxvontwusz.html?
One of my sites has been getting queries from googlebot, on the order of:
example-log:66.249.79.216 - - [06/Apr/2016:15:36:56 -0700] "GET /vvytnoxvontwusz.html HTTP/1.1" 404 15136 "-" "Mozilla/5.0 (...
- 713
1
vote
1
answer
314
views
Allow Google To Bypass Firewall Nginx
So I am looking for a system which essentially returns a 401 for every visitor that doesn't have a certain cookie. I would like to make it so if the visitor/requester is google then it does not return ...
0
votes
1
answer
420
views
Moved website to new server - updated DNS - web crawlers still hitting old site by IP
About ten days ago I moved a site - mostly a Joomla discussion board - to a new server at a different IP address. During a brief scheduled downtime I replicated the content over and completed DNS ...
- 81
1
vote
2
answers
2k
views
block fake google bots [duplicate]
How could I block DDOS attacks with fake Google bots?
I found 2 solutions on the net. But both seems to block also correct google bots.
# Block fake google when it's not coming from their IP range's ...
- 111
0
votes
1
answer
215
views
Google bot cannot read my web site
I am getting from time to time a message from Google bot that it cannot access my web site.
Over the last 24 hours, Googlebot encountered 1 errors while
attempting to retrieve DNS information ...
- 345
8
votes
3
answers
43k
views
AH01797: client denied by server configuration: /usr/share/doc
Since quite a while (over a month now) I see lines like the following in the apache logs:
180.76.15.138 - - [24/Jun/2015:16:13:34 -0400] "GET /manual/de/mod/module-dict.html HTTP/1.1" 403 396 "-" "...
- 407
1
vote
2
answers
97
views
Googlebot requesting pages of 1 site on another site
Problem: Using Prerender.io to index/store pages of one site, I keep getting path requests that only exist on my old site
Example: on Prerender I'll see that Googlebot requested http://www.new-site....
- 159
1
vote
1
answer
1k
views
Trouble filtering googlebot from apache access log
Though it seems like it should be pretty straightforward, I have been unable to configure apache so that googlebot's requests are not stored in the access log. I've tried the following lines:
...
- 123
0
votes
0
answers
270
views
Apache duplicate every GET request made by Googlebot
System: Linux 3.10.47.core2.24
Apache: Most likely version 2.2 (can`t check that)
Server API: Apache 2.0 Handler
Apache API Version: 20051115
In logs requests looks like this:
94.*.*.* - - [26/Nov/...
-4
votes
1
answer
198
views
How Can I Encourage Google to scan New robots.txt File?
I just updated my robots.txt file on a new site; Google Webmaster Tools reports it read my robots.txt 2 days before my last update.
my last robots.txt had a "disallow: all" raw.
Is there any way I ...
- 1
1
vote
1
answer
1k
views
Google-bot trips on a perfectly normal robots.txt, then on a nonexistent robots.txt [duplicate]
I have two domain names pointing to the same virtual server. One of them, http://ilarikaila.com, is a working brochure website I made for a friend. I used the other one, http://teemuleisti.com, to ...
- 123
4
votes
3
answers
5k
views
fail2ban ignoreip DNS host example?
I would like to add ".googlebot.com" to the ignore iplist for fail2ban since the ignoreip explanation mentions DNS host as an accepted input. Is this a proper format?
# "ignoreip" can be an IP ...
- 1,837
3
votes
1
answer
311
views
Why is googlebot requesting robots.txt from my SSH server?
I run ossec on my server and periodically I receive a warning like this:
Received From: myserver->/var/log/auth.log
Rule: 5701 fired (level 8) -> "Possible attack on the ssh server (or version ...
- 796
1
vote
1
answer
313
views
Googlebot cant access my site webmaster tools reply Unreachable robots.txt
When I try to fetch my site as a googlebot in webmaster tools it return Unreachable robots.txt, after investigate I understood google bot can see my server:
tcpdump | grep google
It returns that ...
- 11
2
votes
1
answer
1k
views
Google-Bot fell in love with my 404-page
Every day my access-log looks kind of this:
66.249.78.140 - - [21/Oct/2013:14:37:00 +0200] "GET /robots.txt HTTP/1.1" 200 112 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot....
- 253
0
votes
0
answers
2k
views
nginx serve different html file for googlebot
I have an angular app served through nginx. For googlebot I want to serve a different static html file so that it can index properly, is the following nginx config correct? (I don't want to complicate ...
- 101
2
votes
1
answer
803
views
Nginx Googlebot rewrite rules failing with 404
Our site is based on Angular which makes it almost completely JavaScript based, therefore we need to serve static HTML snapshots to the Googlebot in order for it to crawl us. At the moment, we have ...
- 21
0
votes
1
answer
155
views
apache rewrite syntax
Trying to block Google bot and others from accessing some of my sites. Thing is I have one box that has a ton of virtual host files that do nothing more than do a proxy pass to other servers. I would ...
- 1
0
votes
1
answer
59
views
Googlebot incrementing page id
So here is an example of a hit I'm getting from the googlebot:
66.249.73.171 - - [19/Feb/2013:16:12:39 -0500] "GET /eghm-blah.php?pid=2855 HTTP/1.1" 200 1684 "-" "Mozilla/5.0 (compatible; Googlebot/2....
- 103
1
vote
1
answer
313
views
Googlebot repeatedly looks for files that aren't on my server
I'm hosting a site for a volunteer organization. I've moved the site to WordPress, but it wasn't always that way. I suspect at one point it was hacked badly.
My Apache error log file has grown to ...
- 167
2
votes
1
answer
408
views
Apache : with Googlebot connections, a single process takes all server memory
Following https://serverfault.com/questions/418735/unbelievable-issue-a-single-apache-process-takes-4-gb-of-memory I post as a new question because I was able to identify the fact that it happens when ...
- 648
0
votes
1
answer
762
views
How to fix googlebot Server Connectivity
I get 'Server Connectivity' error at google webmaster tool. I suspect it is because of iptables rules that I've set to counter some DDoS attacks, thugh I'm not sure which rules could be relevant. This ...
- 3,127