We develop websites and we host the QA environment on the same server as the production environment. I want to serve a specific robots.txt for all QA sites but not for the production sites. We have a lot of sites so I do not want to manyallu update hundreds of vhost configuration blocks. The QA sites are easily identifyable from both the domain name and the directory they are in. QA and production sites are grouped in two different directories. All QA sites are hosted under *.qa.mycompany.com.

Example production: host: example.org docroot: /var/www/production/example.org

Example QA: host: example.qa.mycompany.com docroot: /var/www/qa/example.org

Is there any way to configure Apache to serve a robots.txt for all QA sites but not the production sites without having to update all QA vhost configs?

  • robots.txt is simply a file in the document root. You don't need to update any vhost configurations. – Anthony Geoghegan Oct 13 '15 at 8:29

You could add this

<Directory "/var/www/qa/*">
  Redirect permanent robots.txt http://example.com/qa_robots.txt

Than you could put your qa-robots.txt on a public areay on a global host, the Directory-block should to a redirect to it for all /qa/ folders.

Search engines should also follow this redirection.

Instead of redirection you can also add

ErrorDocument 404 "/var/www/qa_robots.txt"

But this will send an 404 errorcode I´m not sure how searchbots reackt on this. I think there could also be a way to do this with mod_rewrite and a matching RewriteCond

  • I read that many robots do not follow redirects when fetching robots.txt, but I'm going to try this with a RewriteRule in the directory block. – Sander Marechal Oct 13 '15 at 12:02

You can do it with mod_alias & If statement, add this to your server conf :

Alias /robots.txt /path/to/robots.txt
Alias /qa_robots.txt /path/to/robots.txt

<If "%{DOCUMENT_ROOT} =~ /(\x2Fvar\x2Fwww\x2Fqa\x2F)/">
RewriteRule ^robots\.txt$  qa_robots.txt

If statement is only available on Apache 2.4, and you tagged your question as 2.2 & 2.4 ... so it may not solve your trouble if you are using Apache 2.2

As you can't escap / in if statement, / are replaced by x2F

I didn't tested it, but it should theoricaly works.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.