20

I have Fail2Ban running on my Centos Server. (Config below)

In my var/log/messages I noticed something really weird:

Jun 19 12:09:32 localhost fail2ban.actions: INFO   [postfix] 114.43.245.205 already banned

I configured Fail2Ban to add the banned IP to iptables.

My jail.conf:

[postfix]

enabled  = true
filter   = postfix
action   = iptables
port     = smtp,ssmtp
filter   = postfix
logpath  = /var/log/maillog
bantime  = 43200
maxretry = 2

My postfix.conf:

[INCLUDES]

before = common.conf

[Definition]
failregex = reject: RCPT from (.*)\[<HOST>\]: 550 5.1.1
            reject: RCPT from (.*)\[<HOST>\]: 450 4.7.1
            reject: RCPT from (.*)\[<HOST>\]: 554 5.7.1
            reject: RCPT from (.*)\[<HOST>\]: (.*)@yahoo.com.tw
ignoreregex =

My question is how can anybody that has already been blocked in iptables still connect to the server?

1
  • Could you add the output of iptables -L -nv to your question? – Ladadadada Jun 19 '14 at 11:19
15

The recidive jail recommended in the other answer here did not fix the issue for me. I eventually fixed this, however, so here's my method in case it helps others.

Fail2ban only blocks over TCP by default. At least with my setup, I noticed the "already banned" message was appearing when bots came back to try the blocked port over UDP instead.

To fix this issue, tell Fail2ban to block the port over all protocols instead of just TCP. You will need to make this change in /etc/fail2ban/jail.conf and in the [Init] section of every action you are using at /etc/fail2ban/action.d/.

Change this:

# Default protocol
protocol = tcp

To:

# Default protocol
protocol = all

Next, I disabled ICMP echo requests so blocked IPs had no way of hitting the server:

  1. nano /etc/sysctl.conf
  2. Add these two lines:

    net.ipv4.icmp_echo_ignore_all = 1  
    net.ipv4.icmp_echo_ignore_broadcasts = 1
    
  3. Exit and save the file.
  4. Run sysctl -p for the change to take effect.

After that, run fail2ban-client reload and you should not see these "already banned" messages any more unless you are spammed by an IP who gets a couple of access attempts before the block takes effect.

Also, it's important to block all ports for every offender rather than the port they were trying to access by using the action iptables-allports in each of the Jails. Otherwise, they may trigger another Jail and end up as "already banned" in the logs.

4
  • 3
    for me it's not very clear...in my /etc/fail2ban/jail.local some filters have action = iptables-multiport[name=apache-myadmin, port="http,https", protocol=tcp] and some not, should I change all of those? Should I change something in /etc/fail2ban/filter.d? – NineCattoRules Mar 10 '16 at 19:16
  • 2
    sorry, but protocol = all is not working, giving an error! – Patrik Laszlo Feb 16 '19 at 16:56
  • 2
    "iptables v1.6.2: multiport needs -p tcp', -p udp', -p udplite', -p sctp' or `-p dccp'" – Patrik Laszlo Feb 16 '19 at 16:58
  • ok, for me the problem was, that the ban was working, but the attacker was using persistent connections so the ban was not in effect right away, as it was still connected and there was no new connection, the only way to do when it was happening restart the mail server – Patrik Laszlo Feb 17 '19 at 11:31
3

If you look at the output of iptables-save, you will see that the fail2ban chains are setup so they evaluate packets according to the rules defined by the filters, for example:

:fail2ban-ssh - [0:0]
-A INPUT -p tcp -A INPUT -p tcp -A INPUT -p tcp -m multiport --dports 22 -j fail2ban-ssh 
-A fail2ban-ssh -j RETURN

Traffic still reaches the server before the other routing rules are applied and the traffic is rejected. fail2ban still sees this initial traffic, and that is why you see the "already banned" messages. Moreover, there's a special filter for recidivists (/etc/fail2ban/filter.d/recidive.conf):

# Fail2Ban filter for repeat bans
#
# This filter monitors the fail2ban log file, and enables you to add long
# time bans for ip addresses that get banned by fail2ban multiple times.
#
# Reasons to use this: block very persistent attackers for a longer time,
# stop receiving email notifications about the same attacker over and
# over again.
#
# This jail is only useful if you set the 'findtime' and 'bantime' parameters
# in jail.conf to a higher value than the other jails. Also, this jail has its
# drawbacks, namely in that it works only with iptables, or if you use a
# different blocking mechanism for this jail versus others (e.g. hostsdeny
# for most jails, and shorewall for this one).

[INCLUDES]

# Read common prefixes. If any customizations available -- read them from
# common.local
before = common.conf

[Definition]

_daemon = fail2ban\.server\.actions

# The name of the jail that this filter is used for. In jail.conf, name the
# jail using this filter 'recidive', or change this line!
_jailname = recidive

failregex = ^(%(__prefix_line)s| %(_daemon)s%(__pid_re)s?:\s+)WARNING\s+\[(?!%(_jailname)s\])(?:.*)\]\s+Ban\s+<HOST>\s*$

[Init]

journalmatch = _SYSTEMD_UNIT=fail2ban.service PRIORITY=4

# Author: Tom Hendrikx, modifications by Amir Caspi
2

This will happen if the IP address you're banning isn't actually the IP address of the client that's connecting to the server. E.g. if your server happens to be behind a load balancer or proxy.

It took me quite a while to figure this out recently. The red herring was that the logs were configured to capture the X-Forwarded-For IP address, instead of the real source address, which in my case was the load balancer.

In this case fail2ban isn't much help, since banning the offending IP would end up blocking all the traffic.

6
  • So what alternative action did you take? – Hassan Baig Apr 22 '17 at 20:09
  • @HassanBaig - none. Fail2ban can't do anything if it's operating behind a load balancer or reverse proxy. – Dale Anderson Apr 23 '17 at 20:46
  • Hmm then what measures would you take against Distributed DoS occuring at the application layer, say an HTTP GET flood? – Hassan Baig Apr 23 '17 at 22:11
  • 1
    @HassanBaig Talk to your hosting provider. Chances are you're not the only person that has encountered the same issue on their systems. – Dale Anderson Apr 24 '17 at 19:48
  • What you basically need is a firewall in front of your load balancer or reverse proxy, and a way to get your ip list from fail2ban, upstream to the firewall. Atleast this is how I plan to solve the problem. – xekon Jan 12 '20 at 12:33
2

If you are using Docker containers to run your application but fail2ban on the host you may run into this issue: https://github.com/fail2ban/fail2ban/issues/2292

Borrowing from the workarounds there, this might be fixable by configuring in you jail.local the line:

[YOUR-JAIL-NAME]
chain = DOCKER-USER
...

Or for Kubernetes

chain = KUBE-FIREWALL

See for more info: https://github.com/fail2ban/fail2ban/issues/2292#issuecomment-593216779

Another option might be in some cases be to use host networking instead of Docker Networking, see: https://docs.docker.com/network/host/

0

I want to contribute my own problem and solution with "already banned" messages. As you wrote I had hundreds of them within minutes while the attacker should have been already banned.

Before I start, here is my system:

  • Plesk 12
  • Centos 7
  • Plesk-Module installed, operating and configuring fail2ban for me

When I installed OpenVPN on my rootserver I had switched firewalld to iptables. That might have caused this problem for me, but aside from that my system was mostly untouched and quite freshly installed (Strato rootserver suggested install image).

If you have this problem, please check /etc/fail2ban/jail.d/00-firewalld.conf for a line that looks like this:

banaction = firewallcmd-ipset

From the time I commented that out, saved the file and restarted fail2ban.service, everything was just fine with fail2ban. No more messages

I am not an expert but hoping to provide you the right answer. If that works for you, please let me know!

0

My question is how can anybody that has already been blocked in iptables still connect to the server?

It connected to the server just one time, but in that one connection it tried to deliver multiple emails to proabably inexistant mailboxes (like info@domain.com, sales@domain.com, tech@domain.com etc.)

You have configured your postfix filter to ban those attempts so the IP will get banned after X tries. The client may already be disconnected from postfix, but since postfix may not have finished processing all its email, fail2ban can detect another attempt from the same client when postfix processes its mail, and thus you get the message address already banned. It's because of how the postfix queue works.

0

My question is how can anybody that has already been blocked in iptables still connect to the server?

Incredible good question. I was searching around if my firewall rules are not working, but the iptables --list-rules matched exactly with another production server with working fail2ban.

The mindblowing solution was to add port 8080 to the blocked ports, as I was still accessing the login page over development ports.

So the fix in my situation is this problem was a quite simple adaption of my jail.local:

[JIRA-LOGIN-tcp]
  enabled = true
  port = http,https,8080
  protocol = tcp
  filter = JIRA-LOGIN-ERROR
  logpath = /var/atlassian/application-data/jira/log/atlassian-jira-security.log
  bantime = 600
  maxretry = 1
0

see https://unix.stackexchange.com/a/525798/22315

you are probably missing port 587 from the "port=" line, and you can check the postfix configuration file or do "lsof -i :587" to find out, directly, if postfix has been configured to open that port.

0

i had this problem, it was just missing ports in the actions parameter

Changing to port="pop3,imap,imaps,smtp,smtps" solved the problem (see files in jail.d directory)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.