1

Trying to block some bad bots using fail2ban reading Apache access log and stuck setting up correct regex. The line I want to match in the access log is:

5.10.83.65 - - [18/Mar/2014:09:06:38 +0400] "GET /catalog/product_compare/
,,/form_key/QLZ6ZkIwX3FWqme3/ HTTP/1.1" 302 522 "-" "
Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"

And the simplified regex I am trying to use:

failregex = ^<HOST>*(AhrefsBot)

But no matter what I try it's not working. Found this post here is what I am trying to achieve badbots


Here is what I ended up doing to block them all without making great big lists, use at your own risk as it might block legitimate visitors if the words match. I used ignoreregex to allow the good ones, and blocked other bots, as long as they call themselves a bot:

failregex = ^<HOST> -.*compatible;.*(Bot|bot)
ignoreregex = (Google|Yandex|Mail|bing)

Now that is far from ideal, however blocks 99% of unwanted scanner bots freeing up the server, quick and dirty.

  • Now thinking about it a bit more - making great big lists is not ideal. Is it possible to block all what has word Bot|bot in it unless it also contains words Google|Mail|Yandex ? – Vlad Mar 20 '14 at 21:01
2

You're almost there, the * isn't doing what you think it is as it matches 0 or more of the previous character try

^<HOST> -.*(AhrefsBot)

for example

fail2ban-regex '5.10.83.65 - - [18/Mar/2014:09:06:38 +0400] "GET /catalog/product_compare/,,/form_key/QLZ6ZkIwX3FWqme3/ HTTP/1.1" 302 522 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"' '^<HOST> -.*(AhrefsBot)'
Running tests
=============

Use   failregex line : ^<HOST> -.*(AhrefsBot)
Use      single line : 5.10.83.65 - - [18/Mar/2014:09:06:38 +0400] "GET /...


Results
=======

Failregex: 1 total
|-  #) [# of hits] regular expression
|   1) [1] ^<HOST> -.*(AhrefsBot)
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [1] Day/MONTH/Year:Hour:Minute:Second
`-

Lines: 1 lines, 0 ignored, 1 matched, 0 missed
  • Great, thank you. Now thinking about it a bit more - making great big lists is not ideal. Is it possible to block all what has word Bot|bot in it unless it also contains words Google|Mail|Yandex ? – Vlad Mar 20 '14 at 21:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.