Can I do the same as robots.txt through IIS settings? Telling

User-agent: * Disallow: / 

in host header or through web.config?

  • 1
    Why don't you want to use a robots.txt file? – dunxd Sep 19 '11 at 9:13
  • Because of deployment issues. – jpkeisala Sep 19 '11 at 10:12
  • "Because of deployment issues" - using an approach which would prevent access (robots.txt is just a polite request) will not be possible by your criteria. The obvious way to prevent access would be to resitrict to a whitelist of IP addresses. But that's at least as difficult to deploy as a robots.txt file – symcbean Sep 19 '11 at 14:26

No this is not a specified way. Either have the robots.txt or the HTML-Meta-Tag as described in http://en.wikipedia.org/wiki/Meta_element#The_robots_attribute

There are only these two options available.


You could use a single robots.txt file stored somewhere on your web server, and redirect all requests to any site hosted on your IIS server for robots.txt to that file. That would be a change per site in IIS, but you may be able to do it programmatically with PowerShell.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.