Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Please can someone explain to me the difference between setting allow and disallow in a robots.txt file and create No follow, No index meta tags! Is it possible to set no follow and no index within the robots.txt file? I have look on and not really got what im looking for! Thanks

share|improve this question

migrated from Aug 20 '09 at 16:40

This question came from our site for professional and enthusiast programmers.

The robots.txt file will simple tell a robot where he is allowed to go and where he isn't. It has no power to make a robot follow a link to another website. The nofollow, noindex meta tag will prevent a compliant robot from indexing or following the link specified or the links contained in the webpage.

share|improve this answer

Disallowing a robot in robots.txt from a specific page has the same effect setting a META tag of noindex for that page does.

A META tag of nofollow prevents search engines from following the links on the page even if they're allowed to index the page itself. There's no robots.txt version of this.

noindex can be useful in some situations on dynamic sites where a robots.txt is not flexible enough. For example, if you had a system that made URLs like node/123, blocking articles based on certain data in the article would have to be done with noindex, as robots.txt can only filter based on stuff in the URL itself.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.