I have a jail that picks up on a specific user agent. It eventually bans it, but it still gets by with several hundred requests.

Here is the relevant info (unless anything else is needed):

Ban action in /etc/fail2ban/jail.conf

banaction = iptables-allports

Here is the entry in /etc/fail2ban/jail.local

enabled = true
filter = domaincrawler-bot
logpath = /var/log/nginx/*access.log
port = 8221,8222,8231,8232
maxretry = 1
findtime = 10
bantime = -1
action = iptables-allports[name=domaincrawler-bot]

I have a shell script that groups together the most frequent hits by IP Address, and it still manages to get hundreds of requests in before banning:

Count    IP Address    User Agent String

543 | "DomainCrawler/3.0 (info@domaincrawler.com; http://www.domaincrawler.com/***************************.com)" "-"
455 | "DomainCrawler/3.0 (info@domaincrawler.com; http://www.domaincrawler.com/********.com)" "-"
282 | "DomainCrawler/3.0 (info@domaincrawler.com; http://www.domaincrawler.com/********************.com)" "-"

I can verify that it does eventually get sent to iptables:

root@****:/var/log/nginx# iptables -L -vn | grep
0     0 REJECT     all  --  *      *            reject-with icmp-port-unreachable

Here is my regex entry in /etc/fail2ban/filter.d/domaincrawler-bot.conf

failregex = ^\d{4} <HOST> .*DomainCrawler.*

Performing a regex test yields thousands of matched lines:

root@****:/var/log/nginx# fail2ban-regex --print-all-matched access.log "^\d{4} <HOST> .*DomainCrawler.*"

 Running tests

Use   failregex line : ^\d{4} <HOST> .*DomainCrawler.*
Use         log file : access.log
Use         encoding : UTF-8


Failregex: 2222 total
|-  #) [# of hits] regular expression
|   1) [2222] ^\d{4} <HOST> .*DomainCrawler.*

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [80276] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?

Lines: 80276 lines, 0 ignored, 2222 matched, 78054 missed [processed in 7.69 sec] 

I also verified on regex101.com that it does find the characters:


And this is the output when checking the jail status:

    root@****:/var/log/nginx# fail2ban-client status domaincrawler-bot
Status for the jail: domaincrawler-bot
|- Filter
|  |- Currently failed: 1
|  |- Total failed: 31178
|  `- File list:    /var/log/nginx/access.log
`- Actions
   |- Currently banned: 12
   |- Total banned: 12
   `- Banned IP list:

This particular server gets a lot of traffic, so perhaps it is lagging behind slightly when parsing the access log? Is there anything else I can do to improve the performance? As I said, it does eventually ban the IP, but not before getting hundreds (sometimes thousands) of requests in for different jails.

Thank you all.

  • What Linux distribution is this? – Michael Hampton May 17 '19 at 17:55
  • @MichaelHampton Ubuntu Server 16.04 – IRGeekSauce May 17 '19 at 19:26

There is an interval of N seconds between log scans. It is something like 1-5 seconds. Fail2ban blocks after maxretry or N seconds, which ever comes last.

  • Perhaps I missed that in the documentation. That makes sense. Thank you so much. I will accept this as the answer. – IRGeekSauce May 20 '19 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.