Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Perishable Press publishes a managed user agent blacklist frequently, which blocks bad bots which htaccess directives.

Article to be found here: http://perishablepress.com/press/2009/03/29/4g-ultimate-user-agent-blacklist/

Would you recommend using such a blacklist? The main goal is to reduce request on my servers, because I see a lot of traffic is consumed by bots visiting my site.

Cheers

share|improve this question
add comment

2 Answers

up vote 0 down vote accepted

Generally yes if it meets your requirements/needs, unless there are specific conflicts you can find with their definitions and/or approaches. The prudent route is to monitor very closely during its initial implementation to ensure it fits your needs without introducing any unanticipated/unexpected side-affects.

share|improve this answer
add comment

You're better off using robots.txt to control the well-behaved bots, and request rate-limiting for evil bots. You can't use anything controlled by the client for badly-behaved bots, because they can modify themselves to evade your filters.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.