1

3 times over the past 48 hours, we had about a 6X spike in traffic lasting a few hours. The server handled it each time, but just barely. This was almost all bad-bot traffic (or maybe failed DOS attempts). I need to set up some sort of wall to automatically block this while it's happening (not by manually going through the logs the next day). Of course, fail2ban comes up a lot in my searching, but I can't find any good examples or documentation to know if it fits for me.

I need to implement a filter for apache that ignores most of the log depending on a regex for both the url requested, as well as the user-agent. And only then starts determining if an IP is hitting us too much.

The problem is that I can't just use a simple threshold ban on ip addresses. Every legitimate page request is immediately followed by lots of other requests for supporting content (GET /images/...; GET /extensions/...; GET /skins/...), so just blocking anyone who made even 10 requests in 2 seconds would catch almost everyone. I need to filter these entries out of any calculations, and count only the top-level requests.

But even if I get it to only look at top-level requests (so now maybe I ban on 20 requests in 10 minutes), then my problem will be that I could easily block crawlers that I'm happy to serve pages to (GoogleBot, Slurp, etc), so I also want it to skip those.

Is Fail2Ban or any other product out there this robust? I can get the info I need rather quickly with a couple of grep -v statements, but is there an existing program that already does this?

Improve this question
3
  • fail2ban works by scanning log files and blocking hosts that generate more than a specified number of lines that match a certain regex expression. You can easily write a regex expression that matches only the URLs and user agents that you're interested in.
    – tlng05
    Jan 7, 2015 at 0:05
  • But can it do a negative regex? Lines to ignore, or through whitelisting somehow? That's what I cannot find anywhere. What I need would be rather complex (if even possible) to write with a readahead regex. It would be at least 10x simpler if I could just say look at every line in the log file, except those that match a certain pattern.
    – Inukshuk
    Jan 7, 2015 at 14:53
  • I found you entirely missing some important information and going toward wrong direction. It may consume your time and energy while your server remains vulnerable to the attackers. But before going towards the solution, I just need to see the web server access log file, Can you please share the log file? it is always difficult to tell what to do without reviewing web server access log file. As per my experience, Fail2Ban limits DDos attack up to some extent but it does not prevent it (please note that DoS attacks are history now, it is very easy to launch a DDos if a site is vulnerable). At th
    – user263859
    Jan 7, 2015 at 20:48

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.