I have a jail that picks up on a specific user agent. It eventually bans it, but it still gets by with several hundred requests.
Here is the relevant info (unless anything else is needed):
Ban action in /etc/fail2ban/jail.conf
banaction = iptables-allports
Here is the entry in /etc/fail2ban/jail.local
[domaincrawler-bot]
enabled = true
filter = domaincrawler-bot
logpath = /var/log/nginx/*access.log
port = 8221,8222,8231,8232
maxretry = 1
findtime = 10
bantime = -1
action = iptables-allports[name=domaincrawler-bot]
I have a shell script that groups together the most frequent hits by IP Address, and it still manages to get hundreds of requests in before banning:
Count IP Address User Agent String
543 80.248.225.168 | "DomainCrawler/3.0 ([email protected]; http://www.domaincrawler.com/***************************.com)" "-"
455 80.248.225.79 | "DomainCrawler/3.0 ([email protected]; http://www.domaincrawler.com/********.com)" "-"
282 80.248.225.4 | "DomainCrawler/3.0 ([email protected]; http://www.domaincrawler.com/********************.com)" "-"
I can verify that it does eventually get sent to iptables:
root@****:/var/log/nginx# iptables -L -vn | grep 80.248.225.4
0 0 REJECT all -- * * 80.248.225.4 0.0.0.0/0 reject-with icmp-port-unreachable
Here is my regex entry in /etc/fail2ban/filter.d/domaincrawler-bot.conf
[Definition]
failregex = ^\d{4} <HOST> .*DomainCrawler.*
Performing a regex test yields thousands of matched lines:
root@****:/var/log/nginx# fail2ban-regex --print-all-matched access.log "^\d{4} <HOST> .*DomainCrawler.*"
Running tests
=============
Use failregex line : ^\d{4} <HOST> .*DomainCrawler.*
Use log file : access.log
Use encoding : UTF-8
Results
=======
Failregex: 2222 total
|- #) [# of hits] regular expression
| 1) [2222] ^\d{4} <HOST> .*DomainCrawler.*
`-
Ignoreregex: 0 total
Date template hits:
|- [# of hits] date format
| [80276] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
`-
Lines: 80276 lines, 0 ignored, 2222 matched, 78054 missed [processed in 7.69 sec]
I also verified on regex101.com that it does find the characters:
And this is the output when checking the jail status:
root@****:/var/log/nginx# fail2ban-client status domaincrawler-bot
Status for the jail: domaincrawler-bot
|- Filter
| |- Currently failed: 1
| |- Total failed: 31178
| `- File list: /var/log/nginx/access.log
`- Actions
|- Currently banned: 12
|- Total banned: 12
`- Banned IP list: 176.74.192.36 176.74.192.40 176.74.192.42 185.6.8.3 185.6.8.7 185.6.8.9 194.68.17.5 80.248.225.142 80.248.225.168 80.248.225.4 80.248.225.7 80.248.225.79
This particular server gets a lot of traffic, so perhaps it is lagging behind slightly when parsing the access log? Is there anything else I can do to improve the performance? As I said, it does eventually ban the IP, but not before getting hundreds (sometimes thousands) of requests in for different jails.
Thank you all.