I've noticed something odd today.

if i go to http://www.google.com/robots.txt, IE11 shows me the contents of google's robot file. However if i go to my site (still in development) using the same browser and point it to robots.txt, IE asks if i want to open or save it to disk, and if i click open, it opens in notepad and not in the browser like google's.

Is there any difference in doing things like google does (obviously there's a difference in the way the file is served, so what are the pro's and con's of either method)? And how do hey manage to do it? I assume AddType/AddHandler have something to do with it but i can't figure out how to do it.

Here's what fiddler gives me for my site's robots.txt file:

HTTP/1.1 200 OK
Date: Thu, 02 Apr 2015 17:16:06 GMT
Server: Apache/2.4.9 (Win64) PHP/5.5.12
X-Powered-By: PHP/5.5.12
Content-Disposition: attachment; filename="robots.txt"
Content-Length: 33
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/plain

User-agent: *
Disallow: /images/
  • The robots don't care, but it might annoy the humans. – Michael Hampton Apr 2 '15 at 17:12
  • It looks like your PHP application is serving robots.txt, rather than your web server. Check your code. – Michael Hampton Apr 2 '15 at 17:22

On "my site" the file is offered for download because of the content disposition header:

Content-Disposition: attachment; filename="robots.txt"

To not download the file but instead render it in the browser - all that's required is to remove that header.

It's of no real consequence whether the file is served with or without that header, as the intended audience (web-cralwers / bots / Not Humans using a browser) almost certainly ignore it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.