2

Hullo,

Typically, if I type into my address bar, "oneofmysites.com/robots.txt", any browser will display the content of robots.txt. As you can see, this is pretty standard behaviour.

I have just one web server which does not. Instead, robots.txt redirects to the default web page (i.e. "thesiteinquestion.com/"). This notable difference (only one of seven sites) worries me.

Questions: Is this something to be concerned about? If so, what is the likely error that I am missing?

Notes:

  • This site is the only one with a separate service provider that I use.
  • CentOS release 6.10 (Final)
  • Webmin
  • robots.txt file permissions are 644
5

It depends on the server configuration, .txt files may not be allowed. It is possible that there is a rule somewhere in the config or some .htaccess that specifies if a url doesn't match a certain pattern (say .html, .php, .htm, etc) it then redirects the rest to the index page of the web root.

  • 2
    Well blue blistering barnacles! You are right. And I did it to myself with this rewrite: RewriteRule \.(gif|jpg|js|txt)$ https://www.thesiteinquestion.com/index.php [L]. I did this to prevent direct access, but I forgot that I added txt files as well. Comment it out, and it works a trice. Question: is there anyway to conditionally exclude files (this robots.txt file, in particular) from a rewrite? – Parapluie Feb 7 at 0:34
  • @Parapluie possibly with a rule that allows robots.txt before the one you have there. I think the webserver would go sequentially through the rules and act on the first match. So if it matches robots\.txt then it will act on that line.Examples here: serverfault.com/questions/213422/… – Serge Rivest Feb 11 at 22:09
1

To add a bit of information, the web provider is not at all forced to respect the robots.txt standard, thus can make what ever he want with it and like Serge told it can be redirected anywhere.

  • The "web provider" is not forced to respect the standard? Am I misunderstanding?: Do you mean the crawler? – Parapluie Feb 7 at 0:35
  • @Parapluie I mean the hoster is not forced to follow the robots.txt standard, and thus crawler must adapt to such case – yagmoth555 Feb 7 at 0:37
  • That is interesting and germane. Thankfully, I have full access to the config in this case (even though my having access was the problem in the first place, at least I can fix it!) Thanks! – Parapluie Feb 7 at 0:40
1

A crawler should read robots.txt and follow its restrictions, but the web server cannot enforce this.

.htaccess (or the server confía file) can be used to keep out crawlers that don’t comply, if you know who they are.

  • Yes, indeed. I am currently using a jail script to ban IPs who ignore the robots.txt directives. i.e. – Parapluie Feb 8 at 16:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.