3 times over the past 48 hours, we had about a 6X spike in traffic lasting a few hours. The server handled it each time, but just barely. This was almost all bad-bot traffic (or maybe failed DOS attempts). I need to set up some sort of wall to automatically block this while it's happening (not by manually going through the logs the next day). Of course, fail2ban comes up a lot in my searching, but I can't find any good examples or documentation to know if it fits for me.
I need to implement a filter for apache that ignores most of the log depending on a regex for both the url requested, as well as the user-agent. And only then starts determining if an IP is hitting us too much.
The problem is that I can't just use a simple threshold ban on ip addresses. Every legitimate page request is immediately followed by lots of other requests for supporting content (GET /images/...; GET /extensions/...; GET /skins/...), so just blocking anyone who made even 10 requests in 2 seconds would catch almost everyone. I need to filter these entries out of any calculations, and count only the top-level requests.
But even if I get it to only look at top-level requests (so now maybe I ban on 20 requests in 10 minutes), then my problem will be that I could easily block crawlers that I'm happy to serve pages to (GoogleBot, Slurp, etc), so I also want it to skip those.
Is Fail2Ban or any other product out there this robust? I can get the info I need rather quickly with a couple of grep -v statements, but is there an existing program that already does this?