Rewrite robots.txt based on host with htaccess

Question

I'm trying to rewrite a filename based on the server's domain.

This code below is wrong / not working, but illustrates the desired effect.

<If "req('Host') != '*.mydevserver.com'">
  RewriteRule "^/robots\.txt$"  "robots-staging.txt"    [R]
</If>

Desired effect is this htaccess will point to a different robots.txt depending on if it's on staging (wildcard subdomain) or not.

Example usage:

http://client1.devsite.com/robots.txt rewritten to
http://client1.devsite.com/robots-staging.txt

http://client2.devsite.com/robots.txt rewritten to
http://client2.devsite.com/robots-staging.txt

https://client.com/robots.txt not rewritten
https://myclient.com/robots.txt

Answer 1

You can use a more dynamic approach:

RewriteRule ^robots\.txt$ robots/%{HTTP_HOST}.txt [NS]

And place your robots.txt files like follows:

• robots/domain.tld.txt
• robots/sub.domain.tld.txt

I came across this solution due to some multi website projects based on TYPO3 CMS and Neos CMS.

Answer 2

I found my own answer. This works as desired.

<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteBase /
    RewriteCond %{HTTP_HOST} ^(.*)?mydevserver(\.com)
    RewriteRule ^robots\.txt$ robots-staging.txt [NS]
</IfModule>