Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Perishable Press publishes a managed user agent blacklist frequently, which blocks bad bots which htaccess directives.

Article to be found here:

Would you recommend using such a blacklist? The main goal is to reduce request on my servers, because I see a lot of traffic is consumed by bots visiting my site.


share|improve this question
up vote 0 down vote accepted

Generally yes if it meets your requirements/needs, unless there are specific conflicts you can find with their definitions and/or approaches. The prudent route is to monitor very closely during its initial implementation to ensure it fits your needs without introducing any unanticipated/unexpected side-affects.

share|improve this answer

You're better off using robots.txt to control the well-behaved bots, and request rate-limiting for evil bots. You can't use anything controlled by the client for badly-behaved bots, because they can modify themselves to evade your filters.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.