Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 1 down vote favorite

Can I do the same as robots.txt through IIS settings? Telling 

User-agent: * Disallow: /

in host header or through web.config?

share|improve this question
1  
Why don't you want to use a robots.txt file? – dunxd Sep 19 '11 at 9:13
Because of deployment issues. – jpkeisala Sep 19 '11 at 10:12
"Because of deployment issues" - using an approach which would prevent access (robots.txt is just a polite request) will not be possible by your criteria. The obvious way to prevent access would be to resitrict to a whitelist of IP addresses. But that's at least as difficult to deploy as a robots.txt file – symcbean Sep 19 '11 at 14:26

2 Answers

up vote 2 down vote accepted

No this is not a specified way. Either have the robots.txt or the HTML-Meta-Tag as described in http://en.wikipedia.org/wiki/Meta_element#The_robots_attribute

There are only these two options available.

share|improve this answer
up vote 0 down vote

You could use a single robots.txt file stored somewhere on your web server, and redirect all requests to any site hosted on your IIS server for robots.txt to that file. That would be a change per site in IIS, but you may be able to do it programmatically with PowerShell.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.