The tag has no wiki summary.

learn more… | top users | synonyms

-4
votes
1answer
24 views

How Can I Encourage Google to scan New robots.txt File?

I just updated my robots.txt file on a new site; Google Webmaster Tools reports it read my robots.txt 2 days before my last update. my last robots.txt had a "disallow: all" raw. Is there any way I ...
0
votes
0answers
7 views

Putting a website in a directory beyond the public_html,Can search engines find it?,Is this website crawled by search engines?,What is the solution? [migrated]

I have a Linux web host and I have added some other domains beside the main domains,as add-ons domains. I have created a directory beyond the main public_html directory, for each of them,and all of ...
1
vote
1answer
80 views

Google-bot trips on a perfectly normal robots.txt, then on a nonexistent robots.txt [duplicate]

I have two domain names pointing to the same virtual server. One of them, http://ilarikaila.com, is a working brochure website I made for a friend. I used the other one, http://teemuleisti.com, to ...
0
votes
0answers
79 views

Googlebot causing 404 for file that exists

I am getting repeated requests, logged in our Apache error logs, from 66.249.76.18 (which appears to be a googlebot), for a URL that is apparently good; the request then gets 404'd and our custom 404 ...
1
vote
1answer
241 views

fail2ban ignoreip DNS host example?

I would like to add ".googlebot.com" to the ignore iplist for fail2ban since the ignoreip explanation mentions DNS host as an accepted input. Is this a proper format? # "ignoreip" can be an IP ...
3
votes
1answer
92 views

Why is googlebot requesting robots.txt from my SSH server?

I run ossec on my server and periodically I receive a warning like this: Received From: myserver->/var/log/auth.log Rule: 5701 fired (level 8) -> "Possible attack on the ssh server (or version ...
1
vote
1answer
110 views

Googlebot cant access my site webmaster tools reply Unreachable robots.txt

When I try to fetch my site as a googlebot in webmaster tools it return Unreachable robots.txt, after investigate I understood google bot can see my server: tcpdump | grep google It returns that ...
2
votes
0answers
155 views

Google-Bot fell in love with my 404-page

Every day my access-log looks kind of this: 66.249.78.140 - - [21/Oct/2013:14:37:00 +0200] "GET /robots.txt HTTP/1.1" 200 112 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; ...
1
vote
0answers
95 views

nginx serve different html file for googlebot

I have an angular app served through nginx. For googlebot I want to serve a different static html file so that it can index properly, is the following nginx config correct? (I don't want to complicate ...
2
votes
1answer
173 views

Nginx Googlebot rewrite rules failing with 404

Our site is based on Angular which makes it almost completely JavaScript based, therefore we need to serve static HTML snapshots to the Googlebot in order for it to crawl us. At the moment, we have ...
0
votes
1answer
55 views

apache rewrite syntax

Trying to block Google bot and others from accessing some of my sites. Thing is I have one box that has a ton of virtual host files that do nothing more than do a proxy pass to other servers. I would ...
0
votes
1answer
41 views

Googlebot incrementing page id

So here is an example of a hit I'm getting from the googlebot: 66.249.73.171 - - [19/Feb/2013:16:12:39 -0500] "GET /eghm-blah.php?pid=2855 HTTP/1.1" 200 1684 "-" "Mozilla/5.0 (compatible; ...
1
vote
1answer
114 views

Googlebot repeatedly looks for files that aren't on my server

I'm hosting a site for a volunteer organization. I've moved the site to WordPress, but it wasn't always that way. I suspect at one point it was hacked badly. My Apache error log file has grown to ...
2
votes
1answer
208 views

Apache : with Googlebot connections, a single process takes all server memory

Following Unbelievable issue : a single apache process takes 4 GB of memory I post as a new question because I was able to identify the fact that it happens when the client connecting is Googlebot. ...
0
votes
1answer
393 views

How to fix googlebot Server Connectivity

I get 'Server Connectivity' error at google webmaster tool. I suspect it is because of iptables rules that I've set to counter some DDoS attacks, thugh I'm not sure which rules could be relevant. This ...