Fail2ban banning odd IP address that are not in logs (0.0.0.4, 0.0.0.5, etc)

Question

Server: Nginx

Fail2ban version: v0.9.3

It seems like no matter what I try, I cannot get fail2ban to find the correct host from the log entry consistently.

/etc/fail2ban/filter/expanse-bot.conf:

[Definition]
failregex = ^(\d{2}|\d{3}) \| <HOST> \| .*\"Expanse indexes the network.*

The ^(\d{2}|\d{3}) catches port 80 or 443. I originally tried a wildcard at the beginning of the line with ^.*<HOST>, but that did not work.

Log entry:

443 | 34.77.162.32 | - | [14/Mar/2021:11:08:23 -0500] | redacted-domain.com | "GET / HTTP/1.1" | 200 | 144126 | "-" | "Expanse indexes the network perimeters of our customers. If you have any questions
 or concerns, please reach out to: [email protected]" | - | 123.45.67.89:1234

In /var/log/fail2ban.log, it is showing these odd IP addresses which do not exist in the log:

2021-03-14 11:07:02,716 fail2ban.actions        [10818]: NOTICE  [expanse-bot] Ban 0.0.0.3
2021-03-14 11:07:03,656 fail2ban.actions        [10818]: NOTICE  [expanse-bot] Ban 0.0.0.4
2021-03-14 11:07:03,865 fail2ban.actions        [10818]: NOTICE  [expanse-bot] Ban 0.0.0.5
2021-03-14 11:07:04,075 fail2ban.actions        [10818]: NOTICE  [expanse-bot] Ban 0.0.0.6

However, then it bans correctly?:

2021-03-14 11:13:48,075 fail2ban.actions        [10818]: NOTICE  [expanse-bot] Ban 34.77.162.13
2021-03-14 11:13:51,288 fail2ban.actions        [10818]: NOTICE  [expanse-bot] Ban 34.77.162.27
2021-03-14 11:15:19,595 fail2ban.actions        [10818]: NOTICE  [expanse-bot] Ban 34.77.162.16
2021-03-14 11:16:30,884 fail2ban.actions        [10818]: NOTICE  [expanse-bot] Ban 34.77.162.12
2021-03-14 11:18:14,208 fail2ban.actions        [10818]: NOTICE  [expanse-bot] Ban 34.77.162.18
2021-03-14 11:19:39,513 fail2ban.actions        [10818]: NOTICE  [expanse-bot] Ban 34.77.162.11

The configuration in my /etc/fail2ban/jail.local:

[expanse-bot]
enabled = true
filter = expanse-bot
logpath = /var/log/nginx/access.log
port = http,https
maxretry = 1
findtime = 3
bantime = 86400
action = iptables-allports[name=expanse-bot]

Once it passes through those odd IP addresses, it does as intended. I just don't understand what the 0.0.0.* means when it's not present in the log? Is there something I am missing in the main configuration?

Answer 1

Although I don't understand how it is possible with anchored failregex like your, I'll try to explain how fail2ban works here.

Tag <HOST> can find hosts also by its hostname. To avoid this you can either use tag <ADDR> instead (>= v.0.10 only) or set usedns = no in jail.

Don't you really have some lines starting with 2-3 digits and pipe and some text hereafter? I mean some multi-line log line...

Fail2ban normally stores the matches to database, so you can also gets the matched lines from there:

sqlite3 'file:/var/lib/fail2ban/fail2ban.sqlite3?mode=ro' "select * from bans where jail = 'expanse-bot' and ip like '0.0.0.%'"

By the way (\d{2}|\d{3}) can be replaced by \d{2,3} and catch-all .* at end does not matter at all and can be safely removed (RE is not anchored at end of message). But .* in the middle (before "Expanse indexes the network") can be replaced by something more precise.

If your RE was unanchored initially (and this odd addresses occurring at restart time) - this may be old, previously matched and banned tickets (fail2ban restores active IPs after restart). Just remove them with manually unbanning, see fail2ban-client --help | grep unban.