Fail2ban bans IP, but not until it already has hundreds of requests

Question

I have a jail that picks up on a specific user agent. It eventually bans it, but it still gets by with several hundred requests.

Here is the relevant info (unless anything else is needed):

Ban action in /etc/fail2ban/jail.conf

banaction = iptables-allports

Here is the entry in /etc/fail2ban/jail.local

[domaincrawler-bot]
enabled = true
filter = domaincrawler-bot
logpath = /var/log/nginx/*access.log
port = 8221,8222,8231,8232
maxretry = 1
findtime = 10
bantime = -1
action = iptables-allports[name=domaincrawler-bot]

I have a shell script that groups together the most frequent hits by IP Address, and it still manages to get hundreds of requests in before banning:

Count    IP Address    User Agent String

543 80.248.225.168 | "DomainCrawler/3.0 (info@domaincrawler.com; http://www.domaincrawler.com/***************************.com)" "-"
455 80.248.225.79 | "DomainCrawler/3.0 (info@domaincrawler.com; http://www.domaincrawler.com/********.com)" "-"
282 80.248.225.4 | "DomainCrawler/3.0 (info@domaincrawler.com; http://www.domaincrawler.com/********************.com)" "-"

I can verify that it does eventually get sent to iptables:

root@****:/var/log/nginx# iptables -L -vn | grep 80.248.225.4
0     0 REJECT     all  --  *      *       80.248.225.4         0.0.0.0/0            reject-with icmp-port-unreachable

Here is my regex entry in /etc/fail2ban/filter.d/domaincrawler-bot.conf

[Definition]
failregex = ^\d{4} <HOST> .*DomainCrawler.*

Performing a regex test yields thousands of matched lines:

root@****:/var/log/nginx# fail2ban-regex --print-all-matched access.log "^\d{4} <HOST> .*DomainCrawler.*"



 Running tests
=============

Use   failregex line : ^\d{4} <HOST> .*DomainCrawler.*
Use         log file : access.log
Use         encoding : UTF-8


Results
=======

Failregex: 2222 total
|-  #) [# of hits] regular expression
|   1) [2222] ^\d{4} <HOST> .*DomainCrawler.*
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [80276] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
`-

Lines: 80276 lines, 0 ignored, 2222 matched, 78054 missed [processed in 7.69 sec] 

I also verified on regex101.com that it does find the characters:

And this is the output when checking the jail status:

    root@****:/var/log/nginx# fail2ban-client status domaincrawler-bot
Status for the jail: domaincrawler-bot
|- Filter
|  |- Currently failed: 1
|  |- Total failed: 31178
|  `- File list:    /var/log/nginx/access.log
`- Actions
   |- Currently banned: 12
   |- Total banned: 12
   `- Banned IP list:   176.74.192.36 176.74.192.40 176.74.192.42 185.6.8.3 185.6.8.7 185.6.8.9 194.68.17.5 80.248.225.142 80.248.225.168 80.248.225.4 80.248.225.7 80.248.225.79

This particular server gets a lot of traffic, so perhaps it is lagging behind slightly when parsing the access log? Is there anything else I can do to improve the performance? As I said, it does eventually ban the IP, but not before getting hundreds (sometimes thousands) of requests in for different jails.

Thank you all.

Answer 1

There is an interval of N seconds between log scans. It is something like 1-5 seconds. Fail2ban blocks after maxretry or N seconds, which ever comes last.