Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

When staging a website on a remote VPS, we would like it to be accessible to team members only, and we would also like to keep the search engine bots off until the site is finalized.

Access control by host whether in Iptables or Apache is not desirable, as accessing hosts can vary.

After some reading in Apache config and other SF postings, I settled on the following design that relies on restricting access to only through specific domain names:

Default virtual host would be disabled in Apache config as follows - relying on Apache behavior to use first virtual host for site default:

<VirtualHost *:80>
# Anything matching this should be silently ignored.

<VirtualHost *:80>
    DocumentRoot /var/www/

<VirtualHost *:80>

Then each team member can add the domain names in their local /etc/hosts:


My question is: is the above technique good enough to achieve the above said goals esp restricting SE bots, or is it possible that bots would work around that.

Note: I understand that mod_rewrite rules con be used to achieve a similar effect as discussed here: How to disable default VirtualHost in apache2?, so the same question would apply to that technique too.

Also please note: the content is not highly secretive - the idea is not to devise something that is hack proof, so we are not concerned about traffic interception or the like. The idea is to keep competitors and casual surfers from viewing the content before it's released, and to prevent SE bots from indexing it.

share|improve this question
up vote 3 down vote accepted

Sure, if you choose a very random domain name, it should be fairly well hidden. However, it would probably be just as easy (and wouldn't require everyone to modify their hosts file) to just throw it behind an http digest authentication prompt. Then just distribute the credentials to your team members, and they'll be able to access it.

In your VirtualHost, just specify a location to protect. This will likely be the same path as your VirtualHost DocumentRoot:

<Location /var/www/>
    AuthName 'Nothing to see here'

    AuthType Digest
    AuthDigestProvider file
    AuthDigestDomain /
    AuthUserFile /path/to/.htdigest

    Require valid-user

Then generate that .htdigest file:

htdigest -c /path/to/.htdigest 'Nothing to see here' 'username'
share|improve this answer
I was going to suggest the same thing, that is some form of simple auth protection. – WerkkreW Apr 6 '12 at 7:25

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.