I have this robots.txt file:

User-agent: *
Sitemap: Path_to_sitemap.xml

My Q is, should I have something else in there as well? Like allow All or something?

Thanks

link|improve this question

feedback

migrated from stackoverflow.com Sep 30 '10 at 11:22

This question came from our site for professional and enthusiast programmers.

2 Answers

up vote 1 down vote accepted

It is a valid robots.txt. You may read this post from google: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

link|improve this answer
feedback

Depends on what you want ;-) You don´t have to allow sth. special, just block the things you wouldn´t have crawled by Google and Co.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown

Not the answer you're looking for? Browse other questions tagged or ask your own question.