We have a custom robots.txt in the root of our IIS cloud service Azure website that does not display correctly when navigating to www.oursite.com/robots.txt . A “different” robots.txt file displays containing:

User-agent: LinkChecker  
Allow: /  
User-agent: *  
Disallow: /  

We’ve tried:
1. Deleting robots.txt file in root – the incorrect robots.txt file still displays.
2. Add url-rewrite rule to web.config to display custom robots.txt – correct robots.txt will display for some time, then revert to different one

Any idea what is stepping on our custom robots.txt?


If you're using IIS management tools for maintenance, it seems that might already have it's own settings which it presumably overwrites - if you've not configured it in the appropriate place:
IIS Robots and Sitemap extension

  • Welcome to Server Fault! Your answer suggests a workable solution to the question is available via another website. The Stack Exchange family of Q&A websites generally frowns on this type of answer because other websites may move, get deleted, or changed. Please read How do I write a good answer? and consider revising your answer to include the steps required to resolve the issue. And don't forget to take the site tour. – Paul Nov 16 '15 at 15:41
  • Uh, well I just provided the resource for extra information. I did answer the question of "Any idea what is stepping on our custom robots.txt?" - it just seemed to me that there might be other possible problems. – Zael Dec 3 '15 at 9:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.