Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Can I do the same as robots.txt through IIS settings? Telling

User-agent: * Disallow: / 

in host header or through web.config?

share|improve this question
Why don't you want to use a robots.txt file? – dunxd Sep 19 '11 at 9:13
Because of deployment issues. – jpkeisala Sep 19 '11 at 10:12
"Because of deployment issues" - using an approach which would prevent access (robots.txt is just a polite request) will not be possible by your criteria. The obvious way to prevent access would be to resitrict to a whitelist of IP addresses. But that's at least as difficult to deploy as a robots.txt file – symcbean Sep 19 '11 at 14:26
up vote 2 down vote accepted

No this is not a specified way. Either have the robots.txt or the HTML-Meta-Tag as described in

There are only these two options available.

share|improve this answer

You could use a single robots.txt file stored somewhere on your web server, and redirect all requests to any site hosted on your IIS server for robots.txt to that file. That would be a change per site in IIS, but you may be able to do it programmatically with PowerShell.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.