Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have the following htaccess file for your consideration:

Options +FollowSymlinks
#+FollowSymLinks must be enabled for any rules to work, this is a security #requirement of the rewrite engine. Normally it's enabled in the root and we #shouldn't have to add it, but it doesn't hurt to do so.

RewriteEngine on
#Apache scans all incoming URL requests, checks for matches in our #.htaccess file 
#and rewrites those matching URLs to whatever we specify.

#allow blank referrers.
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)?site.com [NC]
RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)?site.dev [NC]
RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)?dev.site.com [NC]
RewriteRule \.(jpg|jpeg|png|gif)$ - [NC,F,L]

# if a directory or a file exists, use it directly
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d


# otherwise forward it to index.php
RewriteRule . index.php

# request is for http://dev.site.com
RewriteCond %{HTTP_HOST} ^dev.site.com$ [NC]

# user-agent is a search engine bot
RewriteCond %{HTTP_USER_AGENT} (Googlebot|yahoo|msnbot) [NC]

# return forbidden
RewriteRule ^ - [L,F]

I wish not to make dev.site.com public via google search or similar.

I've place this. Should I wait ? Or is there something else that I should do ?

share|improve this question
    
What's wrong with robots.txt? –  Michael Hampton Oct 18 '12 at 17:12
    
@MichaelHampton are you suggesting that I should use a specific robots.txt for dev.site.com that tells crawlers to not see nothing ? And another robots.txt for site.com ? If I use a robots.txt that enables site.com, wouldn't we enable dev.nuvemk.com as well ? –  MEM Oct 18 '12 at 17:15
add comment

1 Answer 1

up vote 2 down vote accepted

.htaccess isn't really the place to block websites from showing up in Google search indexes. robots.txt is one of the methods designed for this purpose.

Placing this:

User-agent: *
Disallow: /

In a file named "robots.txt" in your root directory of dev.site.com should stop it from appearing.

Alternatively you could include a meta tag in the pages you don't want to appear, such as:

<meta name="robots" content="noindex">

Or if it's only Google that you want to not index your page, but allow others robots you can use:

<meta name="googlebot" content="noindex">

The downside to this in a development/production type scenario is you'll have to do something to make sure those tags don't appear in your production code (assuming you do want Google to index you production server).

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.