Trying to block some bad bots using fail2ban reading Apache access log and stuck setting up correct regex. The line I want to match in the access log is:
5.10.83.65 - - [18/Mar/2014:09:06:38 +0400] "GET /catalog/product_compare/
,,/form_key/QLZ6ZkIwX3FWqme3/ HTTP/1.1" 302 522 "-" "
Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"
And the simplified regex I am trying to use:
failregex = ^<HOST>*(AhrefsBot)
But no matter what I try it's not working. Found this post here is what I am trying to achieve badbots
Here is what I ended up doing to block them all without making great big lists, use at your own risk as it might block legitimate visitors if the words match. I used ignoreregex to allow the good ones, and blocked other bots, as long as they call themselves a bot:
failregex = ^<HOST> -.*compatible;.*(Bot|bot)
ignoreregex = (Google|Yandex|Mail|bing)
Now that is far from ideal, however blocks 99% of unwanted scanner bots freeing up the server, quick and dirty.