Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I would like to setup ACL (python script or any ready made Centos compatible scipt) & limit rate for all IP addresses, who are hitting the site very frequently. For example crawlers, bots (except google & bing bots).

After identifying the ip address, script should automatically add those IP address to IPTABLES so that they cannot access the site.

I have following application added to my Centos OS

  1. Apache 2.x
  2. MySQL 5.2
  3. Varnish
  4. Memcache
  5. APC

I would like the script should satisfy our requirement i.e:

  1. Block crawler/users other than google & bit who are hitting server too hard.
  2. Don't block white listed ip address/ip regions.
  3. Make sure it can be used for commercial purpose.
  4. It doesn't impact performance further.

Please advise what is best possible solution for this.

share|improve this question

You appear to be asking for fail2ban

share|improve this answer
Thanks for providing the link. I have one doubt, will fail2ban block googlebot and bingbot too? I saw someone mentioned this in a forum… – Sukhjinder Singh Dec 11 '12 at 17:50

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.