I'm trying to prevent the search engines from crawling through SSL version of the site, to prevent content duplication and canonization issues.

I found the following great article: http://www.seoworkers.com/seo-articles-tutorials/robots-and-https.html

Problem is, that I'm using lighttpd, which doesn't seem to have the RewriteCond directive, to limit the rewriting only to SSL.

Can anyone advice if this possible on lighttpd, and give a snippet?

Thanks in advance!

link|improve this question

feedback

2 Answers

up vote 1 down vote accepted

try this:

$SERVER["socket"] == "a.b.c.d:443" {
  ...

  $HTTP["host"] =~ "<yourhostname>" {
    url.rewrite-once = ("^robots.txt$" => "robots_ssl.txt")
  }
}

the url.rewrite-once should definitely be in your $SERVER["socket"] section of your ssl part to be only applied when your ssl host is accessed.

url.rewirte-once can be placed without conditional but it seems that it will not work all the time according to some forums.

link|improve this answer
Hi. Thanks for the tip, but unfortunately it didn't work. I still getting the old robots.txt. Any idea? – SyRenity Feb 10 '10 at 18:38
dump questions first ;) have you restarted? when yes, try to vary the regex: url.rewrite-once = ("^/robots.txt$" => "/robots_ssl.txt") – Christian Feb 11 '10 at 5:53
feedback

If you have at least version 1.4.19 you should be able to use conditional configuration, with something like:

$HTTP["scheme"] == "https" {
     url.rewrite-once = ("^robots.txt$" => "robots_ssl.txt")
}
link|improve this answer
Hi. Just tried this - doesn't work, still getting the original robots.txt page. I placed this snippet outside the $SERVER["socket"] block. I use lighttpd 1.4.22, so this shouldn't be a problem. Any suggestions? – SyRenity Jan 31 '10 at 18:23
feedback

Your Answer

 
or
required, but never shown

Not the answer you're looking for? Browse other questions tagged or ask your own question.