Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm trying to prevent the search engines from crawling through SSL version of the site, to prevent content duplication and canonization issues.

I found the following great article:

Problem is, that I'm using lighttpd, which doesn't seem to have the RewriteCond directive, to limit the rewriting only to SSL.

Can anyone advice if this possible on lighttpd, and give a snippet?

Thanks in advance!

share|improve this question
up vote 1 down vote accepted

try this:

$SERVER["socket"] == "a.b.c.d:443" {

  $HTTP["host"] =~ "<yourhostname>" {
    url.rewrite-once = ("^robots.txt$" => "robots_ssl.txt")

the url.rewrite-once should definitely be in your $SERVER["socket"] section of your ssl part to be only applied when your ssl host is accessed.

url.rewirte-once can be placed without conditional but it seems that it will not work all the time according to some forums.

share|improve this answer
Hi. Thanks for the tip, but unfortunately it didn't work. I still getting the old robots.txt. Any idea? – SyRenity Feb 10 '10 at 18:38
dump questions first ;) have you restarted? when yes, try to vary the regex: url.rewrite-once = ("^/robots.txt$" => "/robots_ssl.txt") – Christian Feb 11 '10 at 5:53

If you have at least version 1.4.19 you should be able to use conditional configuration, with something like:

$HTTP["scheme"] == "https" {
     url.rewrite-once = ("^robots.txt$" => "robots_ssl.txt")
share|improve this answer
Hi. Just tried this - doesn't work, still getting the original robots.txt page. I placed this snippet outside the $SERVER["socket"] block. I use lighttpd 1.4.22, so this shouldn't be a problem. Any suggestions? – SyRenity Jan 31 '10 at 18:23

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.