I have this robots.txt file:

User-agent: *
Sitemap: Path_to_sitemap.xml

My Q is, should I have something else in there as well? Like allow All or something?


migrated from stackoverflow.com Sep 30 '10 at 11:22

This question came from our site for professional and enthusiast programmers.


It is a valid robots.txt. You may read this post from google: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449


Depends on what you want ;-) You don´t have to allow sth. special, just block the things you wouldn´t have crawled by Google and Co.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.