Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I know in order to show a directory listing of my files in a browser through .htaccess I can use

Options +Indexes

and to prevent Google and most bots from crawling my directory I can use

Options -Indexes

Is it possible to still allow a visible directory listing through a browser but prevent bot crawling/indexing solely with .htaccess?

share|improve this question
2  
Why are you not using robots.txt? –  tylerl Oct 29 '12 at 5:23
    
It's just a question of curiosity regarding .htaccess. –  RCNeil Oct 29 '12 at 5:46
add comment

1 Answer

up vote 3 down vote accepted

Your .htaccess file cannot magically distinguish "real" users from "bot" users. Since from the webserver's perspective, there is no distinction.

However, as a general rule, bots will respect the contents of robots.txt, while web browsers do not.

Alternately, if you had some way of determining what was a bot and what was not, you could work that rule into your .htaccess configuration. A common tactic is to apply a set of RewriteRules that filter based on the reported User-Agent header. For example, a user-agent that contains the word "googlebot" is probably run by Google.

User-Agents.org has a list of popular user-agent identifiers. But remember that the contents of this header are set by the person running the bot/browser, and can contain anything she wants it to. So, for example, malicious users will typically copy the User-Agent string from a popular browser or perhaps a popular search engine. So you can't depend on this.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.