1

Trying to block some bad bots using fail2ban reading Apache access log and stuck setting up correct regex. The line I want to match in the access log is:

5.10.83.65 - - [18/Mar/2014:09:06:38 +0400] "GET /catalog/product_compare/
,,/form_key/QLZ6ZkIwX3FWqme3/ HTTP/1.1" 302 522 "-" "
Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"

And the simplified regex I am trying to use:

failregex = ^<HOST>*(AhrefsBot)

But no matter what I try it's not working. Found this post here is what I am trying to achieve badbots


Here is what I ended up doing to block them all without making great big lists, use at your own risk as it might block legitimate visitors if the words match. I used ignoreregex to allow the good ones, and blocked other bots, as long as they call themselves a bot:

failregex = ^<HOST> -.*compatible;.*(Bot|bot)
ignoreregex = (Google|Yandex|Mail|bing)

Now that is far from ideal, however blocks 99% of unwanted scanner bots freeing up the server, quick and dirty.

1
  • Now thinking about it a bit more - making great big lists is not ideal. Is it possible to block all what has word Bot|bot in it unless it also contains words Google|Mail|Yandex ?
    – Vlad
    Mar 20, 2014 at 21:01

1 Answer 1

2

You're almost there, the * isn't doing what you think it is as it matches 0 or more of the previous character try

^<HOST> -.*(AhrefsBot)

for example

fail2ban-regex '5.10.83.65 - - [18/Mar/2014:09:06:38 +0400] "GET /catalog/product_compare/,,/form_key/QLZ6ZkIwX3FWqme3/ HTTP/1.1" 302 522 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)"' '^<HOST> -.*(AhrefsBot)'
Running tests
=============

Use   failregex line : ^<HOST> -.*(AhrefsBot)
Use      single line : 5.10.83.65 - - [18/Mar/2014:09:06:38 +0400] "GET /...


Results
=======

Failregex: 1 total
|-  #) [# of hits] regular expression
|   1) [1] ^<HOST> -.*(AhrefsBot)
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [1] Day/MONTH/Year:Hour:Minute:Second
`-

Lines: 1 lines, 0 ignored, 1 matched, 0 missed
1
  • Great, thank you. Now thinking about it a bit more - making great big lists is not ideal. Is it possible to block all what has word Bot|bot in it unless it also contains words Google|Mail|Yandex ?
    – Vlad
    Mar 20, 2014 at 21:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .