Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have this robots.txt file:

User-agent: *
Sitemap: Path_to_sitemap.xml

My Q is, should I have something else in there as well? Like allow All or something?

Thanks

share|improve this question

migrated from stackoverflow.com Sep 30 '10 at 11:22

This question came from our site for professional and enthusiast programmers.

2 Answers 2

up vote 1 down vote accepted

It is a valid robots.txt. You may read this post from google: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

share|improve this answer

Depends on what you want ;-) You don´t have to allow sth. special, just block the things you wouldn´t have crawled by Google and Co.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.