Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have the following htaccess file for your consideration:

Options +FollowSymlinks
#+FollowSymLinks must be enabled for any rules to work, this is a security #requirement of the rewrite engine. Normally it's enabled in the root and we #shouldn't have to add it, but it doesn't hurt to do so.

RewriteEngine on
#Apache scans all incoming URL requests, checks for matches in our #.htaccess file 
#and rewrites those matching URLs to whatever we specify.

#allow blank referrers.
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)? [NC]
RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)? [NC]
RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)? [NC]
RewriteRule \.(jpg|jpeg|png|gif)$ - [NC,F,L]

# if a directory or a file exists, use it directly
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d

# otherwise forward it to index.php
RewriteRule . index.php

# request is for
RewriteCond %{HTTP_HOST} ^$ [NC]

# user-agent is a search engine bot
RewriteCond %{HTTP_USER_AGENT} (Googlebot|yahoo|msnbot) [NC]

# return forbidden
RewriteRule ^ - [L,F]

I wish not to make public via google search or similar.

I've place this. Should I wait ? Or is there something else that I should do ?

share|improve this question
What's wrong with robots.txt? – Michael Hampton Oct 18 '12 at 17:12
@MichaelHampton are you suggesting that I should use a specific robots.txt for that tells crawlers to not see nothing ? And another robots.txt for ? If I use a robots.txt that enables, wouldn't we enable as well ? – MEM Oct 18 '12 at 17:15
up vote 2 down vote accepted

.htaccess isn't really the place to block websites from showing up in Google search indexes. robots.txt is one of the methods designed for this purpose.

Placing this:

User-agent: *
Disallow: /

In a file named "robots.txt" in your root directory of should stop it from appearing.

Alternatively you could include a meta tag in the pages you don't want to appear, such as:

<meta name="robots" content="noindex">

Or if it's only Google that you want to not index your page, but allow others robots you can use:

<meta name="googlebot" content="noindex">

The downside to this in a development/production type scenario is you'll have to do something to make sure those tags don't appear in your production code (assuming you do want Google to index you production server).

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.