Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a redirect rule in my http.conf file to redirect all HTTP requests to HTTPS:

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}

This works well as a blanket approach, however I'd like to exclude robots.txt from being redirected to https (so it should be This is mostly because google refuses to grab my robots.txt over ssl (?)

I've tried a couple of different ways but so far it either 404's the file or ignores it and goes to (as per the rest of the site)

I tried something along the lines of:

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteCond %{REQUEST_URI} !^robots.txt
RewriteRule (.*)$ https://%{HTTP_HOST}/$1 [NC,R=301,L]

Any suggestions on this would be greatly appreciated.

share|improve this question
You can do this, but it may not have the effect you want. Technically, and are different sites. Better to figure out why Google won't fetch your robots.txt and fix that problem. – Michael Hampton Feb 7 '14 at 3:13
A fair point! I'll try and have a look in to this. Logwatch was showing 404 Not Found /robots.txt: 3 Time(s) which triggered my investigation. – Mike Feb 7 '14 at 3:29


RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule ^/robots\.txt$ - [L]
# or if this is in an htaccess or <Directory>:
# RewriteRule ^robots\.txt$ - [L]
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
share|improve this answer
Thanks for the reply, I've tried ... RewriteCond %{HTTPS} off RewriteRule ^/robots\.txt$ - [L] RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} but still the same results as before, I'm afraid – Mike Feb 7 '14 at 2:57
@Mike Where is that configuration located? – Shane Madden Feb 7 '14 at 5:21

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.