Convention to prevent webcrawlers from indexing your website.

learn more… | top users | synonyms

2
votes
1answer
59 views

What's with random-character queries coming from googlebot, e.g., vvytnoxvontwusz.html?

One of my sites has been getting queries from googlebot, on the order of: example-log:66.249.79.216 - - [06/Apr/2016:15:36:56 -0700] "GET /vvytnoxvontwusz.html HTTP/1.1" 404 15136 "-" "Mozilla/5.0 ...
-1
votes
1answer
32 views

Robots.txt - disallow crawling one directory on subdomain

I have placed my product showcase on subdomain such as http://demo.domain.com/productname/. Demo version of product is located at http://demo.domain.com/productname/demo/. I would like to disallow ...
0
votes
0answers
27 views

Remove ?=collcc from url

Today i found at google webmasters tools too many duplicated url, some parameters added i don't know why but i need to remove it, for example http://my-site.com/5454/my-utr.html ...
3
votes
1answer
530 views

Baidu Spider causing 3Gb of traffic a day - but I do business in China

I'm in a difficult situation, the Baidu spider is hitting my site causing about 3Gb a day worth of bandwidth. At the same time I do business in China so don't want to just block it. Has anyone else ...
1
vote
1answer
69 views

Custom robots.txt being overwritten in Azure IIS 8 by something

We have a custom robots.txt in the root of our IIS cloud service Azure website that does not display correctly when navigating to www.oursite.com/robots.txt . A “different” robots.txt file displays ...
0
votes
2answers
76 views

How to serve robots.txt for all my own subdomains but not other hosts on Apache?

We develop websites and we host the QA environment on the same server as the production environment. I want to serve a specific robots.txt for all QA sites but not for the production sites. We have a ...
-3
votes
2answers
40 views

Restricting Access from BOTS

I would like to protect my server from too many hits from Bots. Considering a scenario, where in a server (physical) located in a private network and hitting my server continuously. Do i have a ...
-1
votes
1answer
96 views

Block all bots from .ru domains via robots.txt or htaccess? [closed]

Is there any way to write rules for robots.txt or htaccess that will block all bots that come from a .ru domain? Thanks
0
votes
1answer
67 views

Robots.txt with several VirtualHosts

My web server is running (apache 2.4.10) different virtual hosts for the following domain name : foo.example.com bar.example.com www.example.com example.com Here is the configuration file for my ...
2
votes
1answer
439 views

Nginx robots.txt configuration

I can't seem to properly configure nginx to return robots.txt content. Ideally, I don't need the file and just want to serve text content configured directly in nginx. Here's my config: server { ...
1
vote
0answers
291 views

How to block fake google spider and fake web browser access?

Recently I found that someguys are trying to mirror my website. They are doing this in two ways: Pretend to be google spiders . Access logs are as following: 89.85.93.235 - - [05/May/2015:20:23:16 ...
0
votes
1answer
68 views

Robots file behaviour

I've noticed something odd today. if i go to http://www.google.com/robots.txt, IE11 shows me the contents of google's robot file. However if i go to my site (still in development) using the same ...
0
votes
2answers
118 views

how to write RewriteCond for a url with subdomain

I am trying to ban some bots by writing a RewriteCond rule in htaccess file. Is the following ruleset correct if I add the following: ## block traffic from particular referrers RewriteEngine On ...
0
votes
0answers
37 views

Allocating percent resources for all robots

As many who posted here, my apache server is getting brutally hammerred to near death by robots, most of which are good robots. There is no way to change their crawl rate. Anyone have a suggestion ...
-4
votes
1answer
140 views

How Can I Encourage Google to scan New robots.txt File?

I just updated my robots.txt file on a new site; Google Webmaster Tools reports it read my robots.txt 2 days before my last update. my last robots.txt had a "disallow: all" raw. Is there any way I ...
1
vote
1answer
386 views

apache robots.txt with SSL

I have an .htaccess file with a rewrite rule to get a redirect of every HTTP request to HTTPS. But now I have a problem that my robots.txt is not recognized by some online checker. If I remove the ...
1
vote
1answer
368 views

Google-bot trips on a perfectly normal robots.txt, then on a nonexistent robots.txt [duplicate]

I have two domain names pointing to the same virtual server. One of them, http://ilarikaila.com, is a working brochure website I made for a friend. I used the other one, http://teemuleisti.com, to ...
1
vote
1answer
1k views

Why is my robots.txt not working?

I have this robots.txt: User-Agent: * Disallow: /files/ User-Agent: ia_archiver Allow: / User-agent: Googlebot Disallow: User-agent: googlebot-image Disallow: User-agent: googlebot-mobile ...
1
vote
2answers
599 views

Thousands of robots.txt 404 errors from bots trying to crawl old multisite

Current situation is that we are getting thousands and thousands of 404 errors from bots looking for robots.txt in different places on our site due to domain redirects. Our old website was a ...
3
votes
1answer
153 views

Why is googlebot requesting robots.txt from my SSH server?

I run ossec on my server and periodically I receive a warning like this: Received From: myserver->/var/log/auth.log Rule: 5701 fired (level 8) -> "Possible attack on the ssh server (or version ...
2
votes
3answers
7k views

How to set robots.txt globally in nginx for all virtual hosts

I am trying to set robots.txt for all virtual hosts under nginx http server. I was able to do it in Apache by putting the following in main httpd.conf: <Location "/robots.txt"> SetHandler ...
1
vote
1answer
219 views

Googlebot cant access my site webmaster tools reply Unreachable robots.txt

When I try to fetch my site as a googlebot in webmaster tools it return Unreachable robots.txt, after investigate I understood google bot can see my server: tcpdump | grep google It returns that ...
2
votes
1answer
489 views

Google-Bot fell in love with my 404-page

Every day my access-log looks kind of this: 66.249.78.140 - - [21/Oct/2013:14:37:00 +0200] "GET /robots.txt HTTP/1.1" 200 112 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; ...
2
votes
1answer
113 views

Ideal robots.txt for a gitweb installation? [closed]

I host a few git repositories at git.nomeata.de using gitweb (and gitolite). Occasionally, a search engine spider comes along and begins to hammer the interface. While I generally do want my git ...
0
votes
2answers
680 views

Dynamic robots.txt based on hostname

Is there a way to swap out a robots.txt file in nginx based on hostname? I currently have www.domain.com and backup.domain.com pointing at the same nginx server, but I don't want Google indexing ...
0
votes
1answer
148 views

Is there a chance to block images spiders / bots on dedicated servers without using robots.txt or .htaccess?

We know that we can block certain spiders from crawling websites pages using robots.txt or .htaccess or maybe via the Apache configuration File httpd.conf. But that requires to edit may be a large ...
1
vote
2answers
853 views

Blocking bad bots

I found this script and was wondering if this is just overkill and even worth using? Is it better for me to just use mod_security? # Generated using http://solidshellsecurity.com services # Begin ...
0
votes
1answer
300 views

robots.txt for subdomain iis7

I have two different sites in iis7 both point to the same folder they have different subdomains www.sitename.com foo.sitename.com they are essentially the same website, but it runs different logic ...
0
votes
1answer
76 views

If I redirect all users (Except me) from within .htaccess, do I need a robots.txt file

So.. I have my live version of my site e.g v1.0 at domain.com I then have my development testing version at testing.domain.com I want testing.domain.com to only be accessible to me for testing, and ...
2
votes
3answers
287 views

Block Offline Browsers

Is there a way to block offline browsers (like Teleport Pro, Webzip, etc...) that are showed in the logs as "Mozilla"? Example: Webzip is showed in my site logs as "Mozilla/4.0 (compatible; MSIE 8.0; ...
2
votes
2answers
290 views

IIS Spikes in anonymous users - crippling my server

I have a server running windows server 2008 R2, recently my websites have becoming unresponsive at least once a day, seemingly at random intervals. I have installed some monitoring software and ...
1
vote
0answers
238 views

How to block urls that request for robots.txt in lighttpd?

We have a dedicated development server which runs only test PHP applications on a public network. We have setup session-based authentication for the site. The issue we have is there are lots of 404s ...
1
vote
2answers
895 views

How to create a global robots.txt that gets appended to each domain's own robots.txt on Apache?

I know can create ONE robots.txt file for all domains on an Apache server*, but I want to append to each domain's (if pre-existing) robots.txt. I want some general rules in place for all domains, but ...
1
vote
1answer
93 views

How to Disallow Particular Path in robots.txt

I want to disallow /path but also wanna allow /path/another-path in robots.txt. I already tried: Disallow: /path Or: Disallow: /path$ But doesn't work, I mean it blocked /path/another-path too. ...
-2
votes
2answers
819 views

how to rewrite or redirect old or missing or invalid url to 404 page [duplicate]

Possible Duplicate: Everything You Ever Wanted to Know about Mod_Rewrite Rules but Were Afraid to Ask? I recently upgraded a site and almost all URLs have changed. I have redirected all of ...
5
votes
3answers
4k views

How do I use robots.txt to disallow crawling for only my subdomains?

If I want my main website to on search engines, but none of the subdomains to be, should I just put the "disallow all" robots.txt in the directories of the subdomains? If I do, will my main domain ...
0
votes
1answer
325 views

webcrawler bots load test my website and it fails the test

we run a commerical website with a relavitely few number of customer at any one time ~30 users . Frequently a webcrawler such as google bot, bing bot, and 80legs will bring our site to a grinding ...
1
vote
1answer
61 views

If denying crawlers access to a directory via robots.txt, will it still index a file in that directory if I direct link?

I am denying indexing to a folder called pdf via robots.txt. However, I do direct link to a few files that exist in that directory. Will search engines such as Google index those files, or ignore ...
0
votes
2answers
1k views

How can I use varnish to generate a robots.txt file even for subdomain of the same site?

I want to generate a robots.txt file using Varnish 2.1. That means that domain.com/robots.txt is served using Varnish and also subdomain.domain.com/robots.txt is also served using Varnish. The ...
1
vote
1answer
339 views

Does a forward web proxy exist that checks and obeys robots.txt on remote domains?

Does there exist a forward proxy server that will lookup and obey robots.txt files on remote internet domains and enforce them on behalf of requesters going via the proxy? e.g. Imagine a website at ...
1
vote
2answers
422 views

How to block Spiders / Scrapers by REMOTE hostname / domain for all Virtual Hosts in Apache?

I've seen plenty of robot.txt stuff, and some mod-rewrite solutions that looked promising… but haven't been able to find a simple solution to block Spiders / Scrapers / whoever the hell I want to ...
3
votes
3answers
153 views

How to prevent discovery of a secure URL?

If I have a url that is used for getting messages and I create it like so: http://www.mydomain.com/somelonghash123456etcetc and this URL allows for other services to POST messages to. Is it possible ...
1
vote
2answers
326 views

Is there a way to disallow robots crawling through IIS Management Console for entire site

Can I do the same as robots.txt through IIS settings? Telling User-agent: * Disallow: / in host header or through web.config?
2
votes
2answers
3k views

Is it good idea to ban amazonaws.com [closed]

Site are crawled by anonymous bot hosted on amazon ec2. This robot doesn't respect robots.txt and creates high load on web server so I added check if reverse IP for request ends with "amazonaws.com" ...
2
votes
1answer
76 views

Use robots.txt to prevent crawlers from getting old versions of Trac pages

looking at my Apache access.log I see that crawlers tend to get old versions of pages and documents, like: 119.63.196.86 - - [10/Jun/2011:10:36:31 +0200] "GET /wiki/News?version=14 HTTP/1.1" 200 6073 ...
1
vote
2answers
336 views

Weird entry in access.log on Apache 2.2

I'm running Apache 2.2, and my server runs well. Noticed this weird anomaly in my access.log file, how should I prevent it? robots.txt doesn't seem to be working. 127.0.0.1 - - [17/Apr/2011:12:17:00 ...
0
votes
1answer
147 views

Disallow xml robots.txt

Google webmaster FAQs suggest that this will exclude all xml files from search: User-agent: Googlebot Disallow: /*.xml$ Is this legal for other bots as well? User-agent: * Disallow: /*.xml$
0
votes
1answer
227 views

Cross-submission robots.txt for multiple domains on single host

We are running a site with multiple languages hosted in a single environment on IIS7. For example, oursite.com - english oursite.de - german oursite.es - spanish This is a single-host ...
7
votes
3answers
8k views

How to create robots.txt file for all domains on Apache server

We have a XAMPP Apache development web server setup with virtual hosts and want to stop serps from crawling all our sites. This is easily done with a robots.txt file. However, we'd rather not ...
0
votes
2answers
832 views

301 redirect or disallow on robots.txt?

I recently asked for 301 redirection on ServerFault and I didn't get a proper solution to my problem, but now I have a new idea: use the robots.txt to disallow certain URLs from my site to be ...