Fail2Ban: already banned?

Question

I have Fail2Ban running on my Centos Server. (Config below)

In my var/log/messages I noticed something really weird:

Jun 19 12:09:32 localhost fail2ban.actions: INFO   [postfix] 114.43.245.205 already banned

I configured Fail2Ban to add the banned IP to iptables.

My jail.conf:

[postfix]

enabled  = true
filter   = postfix
action   = iptables
port     = smtp,ssmtp
filter   = postfix
logpath  = /var/log/maillog
bantime  = 43200
maxretry = 2

My postfix.conf:

[INCLUDES]

before = common.conf

[Definition]
failregex = reject: RCPT from (.*)\[<HOST>\]: 550 5.1.1
            reject: RCPT from (.*)\[<HOST>\]: 450 4.7.1
            reject: RCPT from (.*)\[<HOST>\]: 554 5.7.1
            reject: RCPT from (.*)\[<HOST>\]: (.*)@yahoo.com.tw
ignoreregex =

My question is how can anybody that has already been blocked in iptables still connect to the server?

Answer 1

The recidive jail recommended in the other answer here did not fix the issue for me. I eventually fixed this, however, so here's my method in case it helps others.

Fail2ban only blocks over TCP by default. At least with my setup, I noticed the "already banned" message was appearing when bots came back to try the blocked port over UDP instead.

To fix this issue, tell Fail2ban to block the port over all protocols instead of just TCP. You will need to make this change in /etc/fail2ban/jail.conf and in the [Init] section of every action you are using at /etc/fail2ban/action.d/.

Change this:

# Default protocol
protocol = tcp

To:

# Default protocol
protocol = all

Next, I disabled ICMP echo requests so blocked IPs had no way of hitting the server:

1. nano /etc/sysctl.conf

2. Add these two lines:

   net.ipv4.icmp_echo_ignore_all = 1  
   net.ipv4.icmp_echo_ignore_broadcasts = 1
   

3. Exit and save the file.
4. Run sysctl -p for the change to take effect.

After that, run fail2ban-client reload and you should not see these "already banned" messages any more unless you are spammed by an IP who gets a couple of access attempts before the block takes effect.

Also, it's important to block all ports for every offender rather than the port they were trying to access by using the action iptables-allports in each of the Jails. Otherwise, they may trigger another Jail and end up as "already banned" in the logs.

Answer 2

If you are using Docker containers to run your application but fail2ban on the host you may run into this issue: https://github.com/fail2ban/fail2ban/issues/2292

Borrowing from the workarounds there, this might be fixable by configuring in you jail.local the line:

[YOUR-JAIL-NAME]
chain = DOCKER-USER
...

Or for Kubernetes

chain = KUBE-FIREWALL

See for more info: https://github.com/fail2ban/fail2ban/issues/2292#issuecomment-593216779

Another option might be in some cases be to use host networking instead of Docker Networking, see: https://docs.docker.com/network/host/

Answer 3

If you look at the output of iptables-save, you will see that the fail2ban chains are setup so they evaluate packets according to the rules defined by the filters, for example:

:fail2ban-ssh - [0:0]
-A INPUT -p tcp -A INPUT -p tcp -A INPUT -p tcp -m multiport --dports 22 -j fail2ban-ssh 
-A fail2ban-ssh -j RETURN

Traffic still reaches the server before the other routing rules are applied and the traffic is rejected. fail2ban still sees this initial traffic, and that is why you see the "already banned" messages. Moreover, there's a special filter for recidivists (/etc/fail2ban/filter.d/recidive.conf):

# Fail2Ban filter for repeat bans
#
# This filter monitors the fail2ban log file, and enables you to add long
# time bans for ip addresses that get banned by fail2ban multiple times.
#
# Reasons to use this: block very persistent attackers for a longer time,
# stop receiving email notifications about the same attacker over and
# over again.
#
# This jail is only useful if you set the 'findtime' and 'bantime' parameters
# in jail.conf to a higher value than the other jails. Also, this jail has its
# drawbacks, namely in that it works only with iptables, or if you use a
# different blocking mechanism for this jail versus others (e.g. hostsdeny
# for most jails, and shorewall for this one).

[INCLUDES]

# Read common prefixes. If any customizations available -- read them from
# common.local
before = common.conf

[Definition]

_daemon = fail2ban\.server\.actions

# The name of the jail that this filter is used for. In jail.conf, name the
# jail using this filter 'recidive', or change this line!
_jailname = recidive

failregex = ^(%(__prefix_line)s| %(_daemon)s%(__pid_re)s?:\s+)WARNING\s+\[(?!%(_jailname)s\])(?:.*)\]\s+Ban\s+<HOST>\s*$

[Init]

journalmatch = _SYSTEMD_UNIT=fail2ban.service PRIORITY=4

# Author: Tom Hendrikx, modifications by Amir Caspi

Answer 4

I found the possible problem. In the setting, the default is

banaction = iptables-multiport

In [sshd] section, port = ssh.

I have added other ports to ssh. To block all connections, in [DEFAULT]change to:

banaction = iptables-allports
protocol = all

in [sshd] change to

port = all

Answer 5

This will happen if the IP address you're banning isn't actually the IP address of the client that's connecting to the server. E.g. if your server happens to be behind a load balancer or proxy.

It took me quite a while to figure this out recently. The red herring was that the logs were configured to capture the X-Forwarded-For IP address, instead of the real source address, which in my case was the load balancer.

In this case fail2ban isn't much help, since banning the offending IP would end up blocking all the traffic.

Answer 6

I want to contribute my own problem and solution with "already banned" messages. As you wrote I had hundreds of them within minutes while the attacker should have been already banned.

Before I start, here is my system:

• Plesk 12
• Centos 7
• Plesk-Module installed, operating and configuring fail2ban for me

When I installed OpenVPN on my rootserver I had switched firewalld to iptables. That might have caused this problem for me, but aside from that my system was mostly untouched and quite freshly installed (Strato rootserver suggested install image).

If you have this problem, please check /etc/fail2ban/jail.d/00-firewalld.conf for a line that looks like this:

banaction = firewallcmd-ipset

From the time I commented that out, saved the file and restarted fail2ban.service, everything was just fine with fail2ban. No more messages

I am not an expert but hoping to provide you the right answer. If that works for you, please let me know!

Answer 7

My question is how can anybody that has already been blocked in iptables still connect to the server?

It connected to the server just one time, but in that one connection it tried to deliver multiple emails to proabably inexistant mailboxes (like [email protected], [email protected], [email protected] etc.)

You have configured your postfix filter to ban those attempts so the IP will get banned after X tries. The client may already be disconnected from postfix, but since postfix may not have finished processing all its email, fail2ban can detect another attempt from the same client when postfix processes its mail, and thus you get the message address already banned. It's because of how the postfix queue works.

Answer 8

My question is how can anybody that has already been blocked in iptables still connect to the server?

Incredible good question. I was searching around if my firewall rules are not working, but the iptables --list-rules matched exactly with another production server with working fail2ban.

The mindblowing solution was to add port 8080 to the blocked ports, as I was still accessing the login page over development ports.

So the fix in my situation is this problem was a quite simple adaption of my jail.local:

[JIRA-LOGIN-tcp]
  enabled = true
  port = http,https,8080
  protocol = tcp
  filter = JIRA-LOGIN-ERROR
  logpath = /var/atlassian/application-data/jira/log/atlassian-jira-security.log
  bantime = 600
  maxretry = 1

Answer 9

see https://unix.stackexchange.com/a/525798/22315

you are probably missing port 587 from the "port=" line, and you can check the postfix configuration file or do "lsof -i :587" to find out, directly, if postfix has been configured to open that port.

Answer 10

i had this problem, it was just missing ports in the actions parameter

Changing to port="pop3,imap,imaps,smtp,smtps" solved the problem (see files in jail.d directory)

Answer 11

If the issue is in fact a reverse proxy that you can't block the source IP of for fear of blocking legitimate traffic, then perhaps the reverse proxy you're using can block the IP, and you can use Fail2ban to block that address using an API (in some cases). One such case is Cloudflare, which already has Fail2ban actions for such a purpose, using firewall rules, although it may need to be updated since it was initially written using their (now discontinued) API v1 api_json.html endpoint.

See example: https://github.com/fail2ban/fail2ban/blob/master/config/action.d/cloudflare.conf

Answer 12

I added this action to the [postfix-sasl] jail.conf:

 action_ = %(default/action_)s[name=%(__name__)s-tcp, protocol="tcp"]
           %(default/action_)s[name=%(__name__)s-udp, protocol="udp"]

and it worked. Now I see both TCP and UDP in iptables:

# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
f2b-postfix-sasl-udp  udp  --  anywhere             anywhere             multiport dports smtp,igmpv3lite,submission,imap,imaps,pop3,pop3s
f2b-postfix-sasl-tcp  tcp  --  anywhere             anywhere             multiport dports smtp,urd,submission,imap,imaps,pop3,pop3s
DROP       all  -f  anywhere             anywhere
...

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
...

Chain f2b-postfix-sasl-tcp (1 references)
target     prot opt source               destination
REJECT     all  --  srv-141-98-10-159.serveroffer.net  anywhere             reject-with icmp-port-unreachable
RETURN     all  --  anywhere             anywhere

Chain f2b-postfix-sasl-udp (1 references)
target     prot opt source               destination
REJECT     all  --  srv-141-98-10-159.serveroffer.net  anywhere             reject-with icmp-port-unreachable
RETURN     all  --  anywhere             anywhere