Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm trying to prevent the search engines from crawling through SSL version of the site, to prevent content duplication and canonization issues.

I found the following great article: http://www.seoworkers.com/seo-articles-tutorials/robots-and-https.html

Problem is, that I'm using lighttpd, which doesn't seem to have the RewriteCond directive, to limit the rewriting only to SSL.

Can anyone advice if this possible on lighttpd, and give a snippet?

Thanks in advance!

share|improve this question

2 Answers 2

up vote 1 down vote accepted

try this:

$SERVER["socket"] == "a.b.c.d:443" {
  ...

  $HTTP["host"] =~ "<yourhostname>" {
    url.rewrite-once = ("^robots.txt$" => "robots_ssl.txt")
  }
}

the url.rewrite-once should definitely be in your $SERVER["socket"] section of your ssl part to be only applied when your ssl host is accessed.

url.rewirte-once can be placed without conditional but it seems that it will not work all the time according to some forums.

share|improve this answer
    
Hi. Thanks for the tip, but unfortunately it didn't work. I still getting the old robots.txt. Any idea? –  SyRenity Feb 10 '10 at 18:38
    
dump questions first ;) have you restarted? when yes, try to vary the regex: url.rewrite-once = ("^/robots.txt$" => "/robots_ssl.txt") –  Christian Feb 11 '10 at 5:53

If you have at least version 1.4.19 you should be able to use conditional configuration, with something like:

$HTTP["scheme"] == "https" {
     url.rewrite-once = ("^robots.txt$" => "robots_ssl.txt")
}
share|improve this answer
    
Hi. Just tried this - doesn't work, still getting the original robots.txt page. I placed this snippet outside the $SERVER["socket"] block. I use lighttpd 1.4.22, so this shouldn't be a problem. Any suggestions? –  SyRenity Jan 31 '10 at 18:23

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.