I would like to setup ACL (python script or any ready made Centos compatible scipt) & limit rate for all IP addresses, who are hitting the site very frequently. For example crawlers, bots (except google & bing bots).
After identifying the ip address, script should automatically add those IP address to IPTABLES so that they cannot access the site.
I have following application added to my Centos OS
- Apache 2.x
- MySQL 5.2
I would like the script should satisfy our requirement i.e:
- Block crawler/users other than google & bit who are hitting server too hard.
- Don't block white listed ip address/ip regions.
- Make sure it can be used for commercial purpose.
- It doesn't impact performance further.
Please advise what is best possible solution for this.