I am configuring apache2 on debian and would like to allow only robots.txt to be accessed for searching engines, while other .txt files are restricted, I tried to add the followings to .htaccess but no luck:

<Files robots.txt>
Order Allow,Deny
Allow from All

<Files *.txt>
Order Deny,Allow
Deny from All

Can anyone help or give me some hints? I am new comer to apache, thanks a lot.


Use mod_rewrite

RewriteEngine On
RewriteCond %{REQUEST_URI} !/robots\.txt$ [nocase]
RewriteRule \.txt$  -  [forbidden,last]

First, make sure the rewrite engine in enabled.

Next, use a negated-match (!) to apply a conditional to the RewriteRule that excludes any URI's ending in "/robots.txt"

Lastly, if the URI ends in ".txt" then issue a 403 Forbidden.

EDIT: Don't forget the comparison engine is using regex, so you need to escape special characters (ie, .)

  • Thanks! It's worked!! Fully appreciated for your help!!! – user3162764 Aug 19 '14 at 4:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.