Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
5

It depends on the server configuration, .txt files may not be allowed. It is possible that there is a rule somewhere in the config or some .htaccess that specifies if a url doesn't match a certain pattern (say .html, .php, .htm, etc) it then redirects the rest to the index page of the web root.


1

A crawler should read robots.txt and follow its restrictions, but the web server cannot enforce this. .htaccess (or the server confía file) can be used to keep out crawlers that don’t comply, if you know who they are.


1

To add a bit of information, the web provider is not at all forced to respect the robots.txt standard, thus can make what ever he want with it and like Serge told it can be redirected anywhere.


1

There is a proper way with map map $http_host $examplecom_robotstxt { hostnames; default robotstxt/development.txt; example.com robotstxt/production.txt; } server { ... location = /robots.txt { rewrite .* /$examplecom_robotstxt break; } ... } Please note that map definition should be outside of server definition.


Only top voted, non community-wiki answers of a minimum length are eligible