Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [robots.txt]

Convention to prevent webcrawlers from indexing your website.

1
vote
2answers
468 views

How to block Spiders / Scrapers by REMOTE hostname / domain for all Virtual Hosts in Apache?

I've seen plenty of robot.txt stuff, and some mod-rewrite solutions that looked promising… but haven't been able to find a simple solution to block Spiders / Scrapers / whoever the hell I want to ...
0
votes
2answers
546 views

how to write RewriteCond for a url with subdomain

I am trying to ban some bots by writing a RewriteCond rule in htaccess file. Is the following ruleset correct if I add the following: ## block traffic from particular referrers RewriteEngine On ...
1
vote
1answer
209 views

robots.txt route requires a backslash when behind an Application Load Balancer

I have a rails site using an AWS ALB and all routes appear to work except one, robots.txt. I am getting the error "ERR_TOO_MANY_REDIRECTS", link to example: https://www.mamapedia.com/robots.txt ...
1
vote
1answer
589 views

Custom robots.txt being overwritten in Azure IIS 8 by something

We have a custom robots.txt in the root of our IIS cloud service Azure website that does not display correctly when navigating to www.oursite.com/robots.txt . A “different” robots.txt file displays ...
0
votes
1answer
120 views

If I redirect all users (Except me) from within .htaccess, do I need a robots.txt file

So.. I have my live version of my site e.g v1.0 at domain.com I then have my development testing version at testing.domain.com I want testing.domain.com to only be accessible to me for testing, and ...
0
votes
1answer
404 views

Cross-submission robots.txt for multiple domains on single host

We are running a site with multiple languages hosted in a single environment on IIS7. For example, oursite.com - english oursite.de - german oursite.es - spanish This is a single-host environment....
1
vote
0answers
1k views

How to block fake google spider and fake web browser access?

Recently I found that someguys are trying to mirror my website. They are doing this in two ways: Pretend to be google spiders . Access logs are as following: 89.85.93.235 - - [05/May/2015:20:23:16 +...
1
vote
0answers
372 views

How to block urls that request for robots.txt in lighttpd?

We have a dedicated development server which runs only test PHP applications on a public network. We have setup session-based authentication for the site. The issue we have is there are lots of 404s ...
0
votes
0answers
43 views

Allocating percent resources for all robots

As many who posted here, my apache server is getting brutally hammerred to near death by robots, most of which are good robots. There is no way to change their crawl rate. Anyone have a suggestion ...