fail2ban regex not matching

Question

I must admit, regexs have always been a weak spot for me. I've never sat down and spent enough time learning them to use them effectively. However, I really am at a loss as to why this is not matching. I'm trying to make a custom failregex to stop people trying to exploit any xmlrpc.php bugs (and they are constantly brute-forcing in an attempt to do so).

I am using fail2ban v0.9.3, with Apache 2.4.18 on Ubuntu 16.04.2.

Here is my regex:

<HOST> - - \[\d{2}/\w{3}/\d{4}:\d{2}:\d{2}:\d{2} +\d{4}\] "POST /xmlrpc.php HTTP/1.0" 200

And here is an example line that it isn't matching (IP replaced with 0s):

0.0.0.0 - - [06/Apr/2017:07:45:42 +0000] "POST /xmlrpc.php HTTP/1.0" 200 752 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"

I cannot figure out why it's not matching. As far as I can tell, that should be an exact match. Can anyone shed some light?

Additionally, it would be nice to have the time zone offset match for both a + and a - sign... and I think by replacing HTTP/1.0 with HTTP/1.\d{1} it should match both 1.0 and 1.1, correct?

Thanks in advance!

EDIT: After playing around on the commandline with fail2ban-regex, I've figured out that with a regex of <HOST> - - \[ it will match all lines. However, the second I change it to <HOST> - - \[\d{2} it does not match any lines at all. This doesn't seem to make sense!

EDIT 2: Well, in the meantime, I'm using <HOST> - - \[.* "POST /(xmlrpc|wp-login).php HTTP/1.\d" 200 and that is matching the lines I want. However, I would still love to know why my original one isn't matching, as I'd rather use a more specific regex than the broad one I'm using now.

Answer 1

After a couple of iterations with fail2ban-regex, I think what happens is that fail2ban tries to pick the date from the log line itself, before testing it against your filter pattern.

$ line='0.0.0.0 - - [06/Apr/2017:07:45:42 +0000] "POST /xmlrpc.php'

With .* in place of the timestamp, it matches, but as you noticed, explicitly looking for a number at the start of the timestamp doesn't match:

$ ./fail2ban-regex "$line"  '<HOST> - - \[.*\].*POST /xmlrpc' |grep ^Lines
Lines: 1 lines, 0 ignored, 1 matched, 0 missed

$ ./fail2ban-regex "$line"  '<HOST> - - \[\d+.*\].*POST /xmlrpc' |grep ^Lines
Lines: 1 lines, 0 ignored, 0 matched, 1 missed

But check this out:

$ ./fail2ban-regex "$line"  '<HOST> - - \[ \+0000\].*POST /xmlrpc' |grep ^Lines
Lines: 1 lines, 0 ignored, 1 matched, 0 missed

I removed the whole timestamp from the pattern, instead expecting it to be empty (there's just the timezone indicator). It still matches, hinting that fail2ban actually removes the timestamp before testing against the pattern in the configuration.

So, it may be better to just leave that part out of the filter pattern, and match anything in place of the timestamp:

<HOST> - - \[.*?\] "POST /xmlrpc.php HTTP/1.0" 200

As a side note, your original RE has an unquoted + where the timestamp starts, the plus sign means "at least one repetition of the previous element", so you need to escape it.

Answer 2

Try this regex - I tested it with regexr.com and it is definitely being matched:

<HOST> - - \[\d{2}\/\w{3}\/\d{4}(:\d{2}){3} \+0000\] "POST \/xmlrpc.php HTTP\/1.0" 200

One of the issues was not escaping the slashes - this needs to be done.

EDIT (After I saw your edit): I tried simplifying it a bit and came up with this that matches:

failregex = ^<HOST> - - \[.*\] "POST \/xmlrpc.php.*$

It is not exactly what you wanted but it works