Robots.txt - no follow, no index

Question

Please can someone explain to me the difference between setting allow and disallow in a robots.txt file and create No follow, No index meta tags! Is it possible to set no follow and no index within the robots.txt file? I have look on http://www.robotstxt.org/robotstxt.html and not really got what im looking for! Thanks

Answer 1

The robots.txt file will simple tell a robot where he is allowed to go and where he isn't. It has no power to make a robot follow a link to another website. The nofollow, noindex meta tag will prevent a compliant robot from indexing or following the link specified or the links contained in the webpage.

Answer 2

Disallowing a robot in robots.txt from a specific page has the same effect setting a META tag of noindex for that page does.

A META tag of nofollow prevents search engines from following the links on the page even if they're allowed to index the page itself. There's no robots.txt version of this.

noindex can be useful in some situations on dynamic sites where a robots.txt is not flexible enough. For example, if you had a system that made URLs like node/123, blocking articles based on certain data in the article would have to be done with noindex, as robots.txt can only filter based on stuff in the URL itself.