Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Please can someone explain to me the difference between setting allow and disallow in a robots.txt file and create No follow, No index meta tags! Is it possible to set no follow and no index within the robots.txt file? I have look on http://www.robotstxt.org/robotstxt.html and not really got what im looking for! Thanks

share|improve this question

migrated from stackoverflow.com Aug 20 '09 at 16:40

This question came from our site for professional and enthusiast programmers.

add comment

2 Answers 2

The robots.txt file will simple tell a robot where he is allowed to go and where he isn't. It has no power to make a robot follow a link to another website. The nofollow, noindex meta tag will prevent a compliant robot from indexing or following the link specified or the links contained in the webpage.

share|improve this answer
add comment

Disallowing a robot in robots.txt from a specific page has the same effect setting a META tag of noindex for that page does.

A META tag of nofollow prevents search engines from following the links on the page even if they're allowed to index the page itself. There's no robots.txt version of this.

noindex can be useful in some situations on dynamic sites where a robots.txt is not flexible enough. For example, if you had a system that made URLs like node/123, blocking articles based on certain data in the article would have to be done with noindex, as robots.txt can only filter based on stuff in the URL itself.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.