Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I know in order to show a directory listing of my files in a browser through .htaccess I can use

Options +Indexes

and to prevent Google and most bots from crawling my directory I can use

Options -Indexes

Is it possible to still allow a visible directory listing through a browser but prevent bot crawling/indexing solely with .htaccess?

share|improve this question
Why are you not using robots.txt? – tylerl Oct 29 '12 at 5:23
It's just a question of curiosity regarding .htaccess. – RCNeil Oct 29 '12 at 5:46
up vote 3 down vote accepted

Your .htaccess file cannot magically distinguish "real" users from "bot" users. Since from the webserver's perspective, there is no distinction.

However, as a general rule, bots will respect the contents of robots.txt, while web browsers do not.

Alternately, if you had some way of determining what was a bot and what was not, you could work that rule into your .htaccess configuration. A common tactic is to apply a set of RewriteRules that filter based on the reported User-Agent header. For example, a user-agent that contains the word "googlebot" is probably run by Google. has a list of popular user-agent identifiers. But remember that the contents of this header are set by the person running the bot/browser, and can contain anything she wants it to. So, for example, malicious users will typically copy the User-Agent string from a popular browser or perhaps a popular search engine. So you can't depend on this.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.