1
vote
2answers
192 views

Is there a way to disallow robots crawling through IIS Management Console for entire site

Can I do the same as robots.txt through IIS settings? Telling User-agent: * Disallow: / in host header or through web.config?
0
votes
1answer
586 views

301 redirect or disallow on robots.txt?

I recently asked for 301 redirection on ServerFault and I didn't get a proper solution to my problem, but now I have a new idea: use the robots.txt to disallow certain URLs from my site to be ...
0
votes
2answers
89 views

Is this a valid robots.txt file?

I have this robots.txt file: User-agent: * Sitemap: Path_to_sitemap.xml My Q is, should I have something else in there as well? Like allow All or something? Thanks
1
vote
6answers
2k views

What happens if a website does not have a robots.txt file?

If the robots.txt file is missing in the root directory of a website, how are things treated as: the site is not indexed at all the site is indexed without any restrictions It should logically be ...
0
votes
1answer
208 views

seems to block /my-beautiful-sef-url-123

I have robots.txt that looks like this: User-agent: * Disallow: /system/ Disallow: /admin/ Disallow: /index.php The obvious goal has been to prevent all the ugly URLs from being indexed, as they ...