We have a custom robots.txt in the root of our IIS cloud service Azure website that does not display correctly when navigating to www.oursite.com/robots.txt . A “different” robots.txt file displays containing:
User-agent: LinkChecker Allow: / User-agent: * Disallow: /
1. Deleting robots.txt file in root – the incorrect robots.txt file still displays.
2. Add url-rewrite rule to web.config to display custom robots.txt – correct robots.txt will display for some time, then revert to different one
Any idea what is stepping on our custom robots.txt?