.htaccess - how to avoid a subdomain to be seen on google or similar?

Question

I have the following htaccess file for your consideration:

Options +FollowSymlinks
#+FollowSymLinks must be enabled for any rules to work, this is a security #requirement of the rewrite engine. Normally it's enabled in the root and we #shouldn't have to add it, but it doesn't hurt to do so.

RewriteEngine on
#Apache scans all incoming URL requests, checks for matches in our #.htaccess file 
#and rewrites those matching URLs to whatever we specify.

#allow blank referrers.
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)?site.com [NC]
RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)?site.dev [NC]
RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)?dev.site.com [NC]
RewriteRule \.(jpg|jpeg|png|gif)$ - [NC,F,L]

# if a directory or a file exists, use it directly
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d


# otherwise forward it to index.php
RewriteRule . index.php

# request is for http://dev.site.com
RewriteCond %{HTTP_HOST} ^dev.site.com$ [NC]

# user-agent is a search engine bot
RewriteCond %{HTTP_USER_AGENT} (Googlebot|yahoo|msnbot) [NC]

# return forbidden
RewriteRule ^ - [L,F]

I wish not to make dev.site.com public via google search or similar.

I've place this. Should I wait ? Or is there something else that I should do ?

Answer 1

.htaccess isn't really the place to block websites from showing up in Google search indexes. robots.txt is one of the methods designed for this purpose.

Placing this:

User-agent: *
Disallow: /

In a file named "robots.txt" in your root directory of dev.site.com should stop it from appearing.

Alternatively you could include a meta tag in the pages you don't want to appear, such as:

<meta name="robots" content="noindex">

Or if it's only Google that you want to not index your page, but allow others robots you can use:

<meta name="googlebot" content="noindex">

The downside to this in a development/production type scenario is you'll have to do something to make sure those tags don't appear in your production code (assuming you do want Google to index you production server).