12

I have configured fail2ban to monitor a certain pattern of malicious traffic I'm getting and ban IP addresses associated.

Everything seems to be working great -- the regex is matching the pattern appropriately and the problem IP address is getting added to iptables.

However, when I check the Apache logs I'm still getting hits from the IP address that's getting banned. It is as though iptables is not running at all.

So let me share some specifics just to confirm that everything is configured correctly.

First, I'll clear and reload the iptables rules:

$ sudo iptables -F
$ cat /etc/iptables.firewall.rules 
*filter

#  Allow all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT -d 127.0.0.0/8 -j REJECT

#  Accept all established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

#  Allow all outbound traffic - you can modify this to only allow certain traffic
-A OUTPUT -j ACCEPT

#  Allow HTTP and HTTPS connections from anywhere (the normal ports for websites and SSL).
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT

#  Allow SSH connections
#
#  The -dport number should be the same port number you set in sshd_config
#
-A INPUT -p tcp -m state --state NEW --dport 22 -j ACCEPT

#  Allow ping
-A INPUT -p icmp -j ACCEPT

#  Log iptables denied calls
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

#  Drop all other inbound - default deny unless explicitly allowed policy
-A INPUT -j DROP
-A FORWARD -j DROP

COMMIT
$ sudo iptables-restore < /etc/iptables.firewall.rules
$ sudo iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
    0     0 REJECT     all  --  *      *       0.0.0.0/0            127.0.0.0/8          reject-with icmp-port-unreachable
   14  1432 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    1    60 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 5/min burst 5 LOG flags 0 level 7 prefix "iptables denied: "
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   11  1638 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0  

Now, here's what the fail2ban configuration looks like:

$ cat /etc/fail2ban/filter.d/apache-xmlrpc.conf 
[Definition]
failregex = .*:80 <HOST> .*POST .*xmlrpc\.php.*
ignoreregex =
$ cat /etc/fail2ban/jail.local 
[apache-xmlrpc]

enabled  = true
port     = http,https
filter   = apache-xmlrpc
logpath  = /var/log/apache2/other_vhosts_access.log
maxretry = 6
$ fail2ban-regex /var/log/apache2/other_vhosts_access.log /etc/fail2ban/filter.d/apache-xmlrpc.conf 

Running tests
=============

Use regex file : /etc/fail2ban/filter.d/apache-xmlrpc.conf
Use log file   : /var/log/apache2/other_vhosts_access.log


Results
=======

Failregex
|- Regular expressions:
|  [1] .*:80 <HOST> .*POST .*xmlrpc\.php.*
|
`- Number of matches:
   [1] 29 match(es)

Ignoreregex
|- Regular expressions:
|
`- Number of matches:

Summary
=======

Addresses found:
[1]
    80.82.70.239 (Sat Jul 13 02:41:52 2013)
    80.82.70.239 (Sat Jul 13 02:41:53 2013)
    80.82.70.239 (Sat Jul 13 02:41:55 2013)
    80.82.70.239 (Sat Jul 13 02:41:56 2013)
    80.82.70.239 (Sat Jul 13 02:41:57 2013)
    80.82.70.239 (Sat Jul 13 02:41:58 2013)
    80.82.70.239 (Sat Jul 13 02:41:59 2013)
    80.82.70.239 (Sat Jul 13 02:42:00 2013)
    80.82.70.239 (Sat Jul 13 02:42:02 2013)
    80.82.70.239 (Sat Jul 13 02:42:03 2013)
    80.82.70.239 (Sat Jul 13 02:42:04 2013)
    80.82.70.239 (Sat Jul 13 02:42:05 2013)
    80.82.70.239 (Sat Jul 13 02:42:06 2013)
    80.82.70.239 (Sat Jul 13 02:42:07 2013)
    80.82.70.239 (Sat Jul 13 02:42:09 2013)
    80.82.70.239 (Sat Jul 13 02:42:10 2013)
    80.82.70.239 (Sat Jul 13 02:42:11 2013)
    80.82.70.239 (Sat Jul 13 02:42:12 2013)
    80.82.70.239 (Sat Jul 13 02:42:13 2013)
    80.82.70.239 (Sat Jul 13 02:42:15 2013)
    80.82.70.239 (Sat Jul 13 02:42:16 2013)
    80.82.70.239 (Sat Jul 13 02:42:17 2013)
    80.82.70.239 (Sat Jul 13 02:42:18 2013)
    80.82.70.239 (Sat Jul 13 02:42:19 2013)
    80.82.70.239 (Sat Jul 13 02:42:20 2013)
    80.82.70.239 (Sat Jul 13 02:42:22 2013)
    80.82.70.239 (Sat Jul 13 02:42:23 2013)
    80.82.70.239 (Sat Jul 13 02:42:24 2013)
    80.82.70.239 (Sat Jul 13 02:42:25 2013)

Date template hits:
0 hit(s): MONTH Day Hour:Minute:Second
0 hit(s): WEEKDAY MONTH Day Hour:Minute:Second Year
0 hit(s): WEEKDAY MONTH Day Hour:Minute:Second
0 hit(s): Year/Month/Day Hour:Minute:Second
0 hit(s): Day/Month/Year Hour:Minute:Second
0 hit(s): Day/Month/Year Hour:Minute:Second
70 hit(s): Day/MONTH/Year:Hour:Minute:Second
0 hit(s): Month/Day/Year:Hour:Minute:Second
0 hit(s): Year-Month-Day Hour:Minute:Second
0 hit(s): Year.Month.Day Hour:Minute:Second
0 hit(s): Day-MONTH-Year Hour:Minute:Second[.Millisecond]
0 hit(s): Day-Month-Year Hour:Minute:Second
0 hit(s): TAI64N
0 hit(s): Epoch
0 hit(s): ISO 8601
0 hit(s): Hour:Minute:Second
0 hit(s): <Month/Day/Year@Hour:Minute:Second>

Success, the total number of match is 29

However, look at the above section 'Running tests' which could contain important
information.

As you can see, I have a failregex set up in a filter and the filter is enabled. Using fail2ban-regex, the filter does find a match in the log file I'm monitoring. (I'm actively being hit by a problematic IP address right now which is making testing quite easy.)

So now I restart fail2ban and observe the rules taking effect:

$ sudo service fail2ban restart
 * Restarting authentication failure monitor fail2ban                                                                                                                         [ OK ] 
$ tail /var/log/fail2ban.log -n 50
2013-07-13 02:42:58,014 fail2ban.server : INFO   Stopping all jails
2013-07-13 02:42:58,745 fail2ban.jail   : INFO   Jail 'apache-xmlrpc' stopped
2013-07-13 02:42:59,439 fail2ban.jail   : INFO   Jail 'ssh' stopped
2013-07-13 02:42:59,440 fail2ban.server : INFO   Exiting Fail2ban
2013-07-13 02:43:08,055 fail2ban.server : INFO   Changed logging target to /var/log/fail2ban.log for Fail2ban v0.8.6
2013-07-13 02:43:08,057 fail2ban.jail   : INFO   Creating new jail 'ssh'
2013-07-13 02:43:08,111 fail2ban.jail   : INFO   Jail 'ssh' uses Gamin
2013-07-13 02:43:08,397 fail2ban.filter : INFO   Added logfile = /var/log/auth.log
2013-07-13 02:43:08,404 fail2ban.filter : INFO   Set maxRetry = 6
2013-07-13 02:43:08,414 fail2ban.filter : INFO   Set findtime = 600
2013-07-13 02:43:08,435 fail2ban.actions: INFO   Set banTime = 600
2013-07-13 02:43:09,277 fail2ban.jail   : INFO   Creating new jail 'apache-xmlrpc'
2013-07-13 02:43:09,277 fail2ban.jail   : INFO   Jail 'apache-xmlrpc' uses Gamin
2013-07-13 02:43:09,283 fail2ban.filter : INFO   Added logfile = /var/log/apache2/other_vhosts_access.log
2013-07-13 02:43:09,286 fail2ban.filter : INFO   Set maxRetry = 6
2013-07-13 02:43:09,289 fail2ban.filter : INFO   Set findtime = 600
2013-07-13 02:43:09,292 fail2ban.actions: INFO   Set banTime = 600
2013-07-13 02:43:09,458 fail2ban.jail   : INFO   Jail 'ssh' started
2013-07-13 02:43:09,792 fail2ban.jail   : INFO   Jail 'apache-xmlrpc' started
2013-07-13 02:43:11,361 fail2ban.actions: WARNING [apache-xmlrpc] Ban 80.82.70.239
$ sudo iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  244 39277 fail2ban-apache-xmlrpc  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 80,443
  101  7716 fail2ban-ssh  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 22
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
    0     0 REJECT     all  --  *      *       0.0.0.0/0            127.0.0.0/8          reject-with icmp-port-unreachable
 3404  582K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
  349 20900 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443
   12   720 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
    2    80 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 5/min burst 5 LOG flags 0 level 7 prefix "iptables denied: "
    2    80 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 3331 4393K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain fail2ban-apache-xmlrpc (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      *       80.82.70.239         0.0.0.0/0           
  244 39277 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain fail2ban-ssh (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      *       223.4.147.8          0.0.0.0/0           
  101  7716 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0  

As the fail2ban log shows, the rule set appears to be configured correctly. You can see already that the problematic IP address is being caught right away and banned. The output of iptables shows that it is in fact being dropped.

Already however, I'm observing that there are no dropped packets for that IP address that matches under the fail2ban-apache-xmlrpc chain. Sure enough, I check the apache logs:

$ tail /var/log/apache2/other_vhosts_access.log
www.--SNIP--.com:80 80.82.70.239 - - [13/Jul/2013:02:43:53 +0000] "POST /xmlrpc.php HTTP/1.1" 403 474 "-" "-"
www.--SNIP--.com:80 80.82.70.239 - - [13/Jul/2013:02:43:54 +0000] "POST /xmlrpc.php HTTP/1.1" 403 474 "-" "-"
www.--SNIP--.com:80 80.82.70.239 - - [13/Jul/2013:02:43:56 +0000] "POST /xmlrpc.php HTTP/1.1" 403 474 "-" "-"
www.--SNIP--.com:80 80.82.70.239 - - [13/Jul/2013:02:43:57 +0000] "POST /xmlrpc.php HTTP/1.1" 403 474 "-" "-"
www.--SNIP--.com:80 80.82.70.239 - - [13/Jul/2013:02:43:58 +0000] "POST /xmlrpc.php HTTP/1.1" 403 474 "-" "-"
www.--SNIP--.com:80 80.82.70.239 - - [13/Jul/2013:02:43:59 +0000] "POST /xmlrpc.php HTTP/1.1" 403 474 "-" "-"
www.--SNIP--.com:80 80.82.70.239 - - [13/Jul/2013:02:44:00 +0000] "POST /xmlrpc.php HTTP/1.1" 403 474 "-" "-"
www.--SNIP--.com:80 80.82.70.239 - - [13/Jul/2013:02:44:02 +0000] "POST /xmlrpc.php HTTP/1.1" 403 474 "-" "-"

Nope, it's not getting blocked! I can also confirm this in the fail2ban log:

$ tail /var/log/fail2ban.log
2013-07-13 02:52:30,757 fail2ban.actions: WARNING [apache-xmlrpc] 80.82.70.239 already banned
2013-07-13 02:52:37,767 fail2ban.actions: WARNING [apache-xmlrpc] 80.82.70.239 already banned
2013-07-13 02:52:44,783 fail2ban.actions: WARNING [apache-xmlrpc] 80.82.70.239 already banned
2013-07-13 02:52:51,814 fail2ban.actions: WARNING [apache-xmlrpc] 80.82.70.239 already banned
2013-07-13 02:52:58,830 fail2ban.actions: WARNING [apache-xmlrpc] 80.82.70.239 already banned
2013-07-13 02:53:05,842 fail2ban.actions: WARNING [apache-xmlrpc] 80.82.70.239 already banned
2013-07-13 02:53:11,858 fail2ban.actions: WARNING [apache-xmlrpc] Unban 80.82.70.239
2013-07-13 02:53:12,910 fail2ban.actions: WARNING [apache-xmlrpc] Ban 80.82.70.239
2013-07-13 02:53:20,118 fail2ban.actions: WARNING [apache-xmlrpc] 80.82.70.239 already banned
2013-07-13 02:53:27,129 fail2ban.actions: WARNING [apache-xmlrpc] 80.82.70.239 already banned

It keeps reappearing in the apache log and thus fail2ban is trying to keep banning it!

I quite honestly cannot figure out for the life of me why iptables is not dropping the traffic from this IP address. The rule order seems correct to me, with the DROP coming before anything else.

I've Google a bunch of results where people are having a similar issue, but it always seems to come back to a problem banning SSH traffic where they are on a nonstandard port. In my case I'm just trying to ban an IP address on standard http port 80.

I hope I'm just overlooking something insanely simple. This is a VPS running Ubuntu 12.04 on Linode. If anyone has any ideas, please let me know. Many thanks...

EDIT: Here is the output of iptables -S

$ sudo iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N fail2ban-apache-xmlrpc
-N fail2ban-ssh
-A INPUT -p tcp -m multiport --dports 80,443 -j fail2ban-apache-xmlrpc
-A INPUT -p tcp -m multiport --dports 22 -j fail2ban-ssh
-A INPUT -i lo -j ACCEPT
-A INPUT -d 127.0.0.0/8 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7
-A INPUT -j DROP
-A FORWARD -j DROP
-A OUTPUT -j ACCEPT
-A fail2ban-apache-xmlrpc -s 80.82.70.239/32 -j DROP
-A fail2ban-apache-xmlrpc -j RETURN
-A fail2ban-ssh -s 223.4.147.8/32 -j DROP
-A fail2ban-ssh -j RETURN
4
  • The output of iptables -s may be more helpful to us than the format of iptables -L Jul 13, 2013 at 3:34
  • @lVlint67 - I edited my question to show the output of iptables -S for you. Let me know if that gives you any further insight.
    – jsdalton
    Jul 13, 2013 at 3:38
  • The last Apache log entry is at 02:44:02 and the first fail2ban message that the IP has been banned is at 02:52:30, so I don't see any evidence of it not working. Please provide the full fail2ban.log for the period.
    – mgorven
    Jul 13, 2013 at 3:55
  • The first ban I see from the fail2ban log was at 2:43 but I also noticed the time discrepancies. NOTE: the ban occurs in the So now I restart fail2ban and observe the rules taking effect: block Jul 13, 2013 at 3:59

3 Answers 3

11

The iptables -s output looks correct and I don't know how 80.82.70.239/32 is getting to any:80 on your server through the firewall. My first guess is that you have a proxy / load balancer in front of the server and Apache is logging the HTTP_X_FORWARDED_FOR header or what ever that is called. If that turns out to be the case you will have to move your firewall logic to the proxy / load balancer or down to the application level (Apache mathing the FORWARDED_FOR header and deny access.


Either way:

The next course of action that I would take, is to capture the output of iptables -s you posted above. Disable fail2ban and load the configuration with the fail2ban chain and IP address blocked into iptables.

But do so with the following as the first -A rule:

-A INPUT -p tcp --dport 80 -j LOG --log-prefix "HTTP: "

If you would feel better trapping 80 and 443 go for it. The thought is that the logs from the FIREWALL may show something we are missing if we pay attention to the packets from suspect sources.

4
  • 4
    You nailed it. I don't know why I didn't think of this before. Apache is running behind Cloudflare and I do have Apache configured to log the HTTP_FORWARDED_FOR IP address. The actual traffic is coming in via the Cloudflare server -- so trying to block it using fail2ban / iptables is not going to work. Thank you!
    – jsdalton
    Jul 13, 2013 at 4:07
  • We use haproxy load balancers on campus to handle LB and HA so I have become familiar with the little 'quirks' the setup brings. I don't know what Cloudflare is, but if you have proper access, moving the ban to the load balancer may possible. ... If that is not possible, Apache may be able to handle the blocking with some method but I am not familiar with it off the top of my head. Jul 13, 2013 at 4:15
  • @jsdalton What was the solution you ended up with. In a similiar boat myself right now.
    – Jay
    Jun 7, 2017 at 19:30
  • @jay Since you have to examine http headers you are looking at something called "Layer 7" filtering or firewall. Otherwise you can try to block the attempts at the load balancer/proxy if that is under your control. Jun 15, 2017 at 17:42
1

The output of iptables actually shows that while there is a rule for the IP address fail2ban feels should be filtered and dropped, no packets have gone through the fail2ban xmlrpc chain and actually been dropped by that rule. Instead, all 224 packets that have gone through that chain have been accepted.

That said, the rules are indeed correct. However, more traffic appears to have been accepted by your accept TCP port 80 rule than has been through the fail2ban filter chain. The most probable reason is that the traffic you wanted blocked came in while the fail2ban chain was not yet inserted in input (I notice you don't have it in your default rules, which is probably OK, but it means that if you reload iptables the fail2ban chain won't be in effect immediately).

Try running iptables -z to zero the packet counts and observe the output of iptables -nvL again. The output should not be the same. Additionally, consider saving the rules for the fail2ban chains in the initial rules for iptables (/etc/iptables.firewall.rules). Save the delegation rules like this one:

fail2ban-apache-xmlrpc  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 80,443

Also save the existence of the chains (like fail2ban-apache-xmlrpc), but do not save the actual banned IPs.

1
  • Thanks Falcon. That's likely because I flushed the iptables rules and then it was a few moments before I restarted fail2ban and the fail2ban-apache-xmlrpc rules were added. Nevertheless, I tried zeroing the packet counts and rechecking. The result was pretty much the same. All of the packets going in to the fail2ban-apache-xmlrpc rules are being RETURNed and none are being DROPed. I confirmed that my Apache logs are showing traffic coming in still on that IP address which it should be dropping.
    – jsdalton
    Jul 13, 2013 at 3:33
1

I had exactly the same problem as yours on my own web site. Very similar setup, LAMP stack, a couple of functional fail2ban jails, yet I still saw those supposedly banned IP address showing up in access log file. I don't have any proxy/load balancer in front of Apache.

The solution to my problem was surprisingly simple: move the banning statements right on top of the iptables configuration file!

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .