Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

We have a XAMPP Apache development web server setup with virtual hosts and want to stop serps from crawling all our sites. This is easily done with a robots.txt file. However, we'd rather not include a disallow robots.txt in every vhost and then have to remove it when we went live with the site on another server.

Is there a way with an apache config file to rewrite all requests to robots.txt on all vhosts to a single robots.txt file?

If so, could you give me an example? I think it would be something like this:

RewriteEngine On
RewriteRule  .*robots\.txt$         C:\xampp\vhosts\override-robots.txt [L] 

Thanks!

share|improve this question
    
robots.txt is not mandatory and some crawlers will ignore it. It should not be seen as a security feature. If you want to hide your site until it is ready for public, add authentication. –  Mircea Vutcovici Dec 16 '10 at 19:21

3 Answers 3

up vote 15 down vote accepted

Apache mod_alias is designed for this and available from the core Apache system, and can be set in one place with almost no processing overhead, unlike mod_rewrite.

Alias /robots.txt C:/xampp/vhosts/override-robots.txt

With that line in the apache2.conf file, outside all the vhost's, http://example.com/robots.txt - on any website it serves, will output the given file.

share|improve this answer
    
This. Put an Alias in each <VirtualHost> block. +1. –  Steven Monday Dec 16 '10 at 21:01
    
Thanks! That worked perfectly. I knew there was an easy solution... –  Michael Berkompas Dec 16 '10 at 21:04
    
If you want it on every single virtual-host, you don't need to put it into all of them. It can go on a global level, like the default /manual alias does out of the box. –  Alister Bulman Dec 16 '10 at 21:52
1  
Man, this saved my day, thanks! –  Tomasz Struczyński Mar 16 '12 at 8:28
    
Thanks for the solution although seeing C:/ in there makes me sick to my stomach knowing there is another windows server out there :) I put my edit in my modules.conf file or in mods-enabled/alias.conf like so: Alias /robots.txt /var/www/robots.txt –  unc0nnected Oct 11 '12 at 16:53

Put your common global robots.txt file somewhere in your server's filesystem that is accessible to the apache process. For the sake of illustration, I'll assume it's at /srv/robots.txt.

Then, to set up mod_rewrite to serve that file to clients who request it, put the following rules into each vhost's <VirtualHost> config block:

RewriteEngine on
RewriteRule ^/robots.txt$ /srv/robots.txt [NC, L]

If you're putting the rewrite rules into per-directory .htaccess files rather than <VirtualHost> blocks, you will need to modify the rules slightly:

RewriteEngine on
RewriteBase /
RewriteRule ^robots.txt$ /srv/robots.txt [NC, L]
share|improve this answer
    
Could you explain this "Put your common global robots.txt file somewhere in your server's filesystem that is accessible to the apache process. For the sake of illustration, I'll assume it's at /srv/robots.txt." in more detail? I need to know what you mean by creating a directory available to the apache process? –  Michael Berkompas Dec 16 '10 at 20:55
    
Each site is contained in a folder like testsite.int.devcsd.com under C:\xampp\vhosts –  Michael Berkompas Dec 16 '10 at 20:56
    
@Michael: Don't bother with this overly complicated mod_rewrite hack. Use Alias instead, as suggested by Alister. –  Steven Monday Dec 16 '10 at 21:02

Not sure if you're running XAMPP on Linux or not, but if you are, you could create a symlink from all virtual hosts to the same robots.txt file, but you need to make sure that your Apache configuration for each virtual host is allowed to follow symlinks (under the <Directory> directive's Options FollowSymLinks).

share|improve this answer
    
I'd rather not have to edit every single vhost declaration. There are over 30... Plus, I want it to be an automatic over ride so that I don't have to do anything when a create a new vhost. –  Michael Berkompas Dec 16 '10 at 20:46
    
Michael, just use sed to do a mass edit, pretty easy stuff, you definitely don't need to do it manually. Laid out how to do it here at the bottom: blog.netflowdevelopments.com/2012/10/11/… –  unc0nnected Oct 11 '12 at 17:40

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.