I have this robots.txt file:
User-agent: *
Sitemap: Path_to_sitemap.xml
My Q is, should I have something else in there as well? Like allow All or something?
Thanks
|
I have this robots.txt file:
My Q is, should I have something else in there as well? Like allow All or something? Thanks
| |||
|
feedback
|
This question came from our site for professional and enthusiast programmers.
|
It is a valid robots.txt. You may read this post from google: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449 | |||
|
feedback
|
|
Depends on what you want ;-) You don´t have to allow sth. special, just block the things you wouldn´t have crawled by Google and Co. | |||
|
feedback
|