Please can someone explain to me the difference between setting allow and disallow in a robots.txt file and create No follow, No index meta tags! Is it possible to set no follow and no index within the robots.txt file? I have look on http://www.robotstxt.org/robotstxt.html and not really got what im looking for! Thanks
The robots.txt file will simple tell a robot where he is allowed to go and where he isn't. It has no power to make a robot follow a link to another website. The nofollow, noindex meta tag will prevent a compliant robot from indexing or following the link specified or the links contained in the webpage.
Disallowing a robot in
robots.txt from a specific page has the same effect setting a
META tag of
noindex for that page does.
META tag of
nofollow prevents search engines from following the links on the page even if they're allowed to index the page itself. There's no
robots.txt version of this.
noindex can be useful in some situations on dynamic sites where a
robots.txt is not flexible enough. For example, if you had a system that made URLs like
node/123, blocking articles based on certain data in the article would have to be done with
robots.txt can only filter based on stuff in the URL itself.