Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have a redirect rule in my http.conf file to redirect all HTTP requests to HTTPS:

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}

This works well as a blanket approach, however I'd like to exclude robots.txt from being redirected to https (so it should be http://www.mysite.com/robots.txt) This is mostly because google refuses to grab my robots.txt over ssl (?)

I've tried a couple of different ways but so far it either 404's the file or ignores it and goes to https://www.mysite.com/robots.txt (as per the rest of the site)

I tried something along the lines of:

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteCond %{REQUEST_URI} !^robots.txt
RewriteRule (.*)$ https://%{HTTP_HOST}/$1 [NC,R=301,L]

Any suggestions on this would be greatly appreciated.

share|improve this question
1  
You can do this, but it may not have the effect you want. Technically, http://www.example.com/ and https://www.example.com/ are different sites. Better to figure out why Google won't fetch your robots.txt and fix that problem. –  Michael Hampton Feb 7 '14 at 3:13
    
A fair point! I'll try and have a look in to this. Logwatch was showing 404 Not Found /robots.txt: 3 Time(s) which triggered my investigation. –  Mike Feb 7 '14 at 3:29

1 Answer 1

Sure.

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule ^/robots\.txt$ - [L]
# or if this is in an htaccess or <Directory>:
# RewriteRule ^robots\.txt$ - [L]
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
share|improve this answer
    
Thanks for the reply, I've tried ... RewriteCond %{HTTPS} off RewriteRule ^/robots\.txt$ - [L] RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} but still the same results as before, I'm afraid –  Mike Feb 7 '14 at 2:57
    
@Mike Where is that configuration located? –  Shane Madden Feb 7 '14 at 5:21

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.