Today MySQL was killed off by the oom-killer on one of our shared web servers, which caused an outage for our customers. When I investigated what happened I found there were over 1,300 requests in the space of 20 seconds from one IP address. They hit every vhost on the server which meant PHP-FPM had to be spun up for every single site, which in turn led to the OOM situation. I'm going to be adjusting the oom-killer to avoid having it kill MySQL in the future, but that's not what this question is about.
What I want to know is whether there is a sensible way to detect this situation in the future. No bot needs to be making >66 requests per second, especially not across multiple vhosts. It feels like it ought to be possible to detect large, sudden spikes in traffic going to multiple vhosts, and take appropriate action to stop them.
We already implement fail2ban but this doesn't feel like it truly falls under its purview, and I certainly can't see a sane way of implementing a check using that tool which wouldn't run the risk of blocking legitimate spiders that cross vhosts without causing issues (e.g. Google Bot and Bing Bot). We also use Varnish in front of the web server to ease the burden on Apache, and from there we block known bad bot UAs and other tells from bots. But this bot identified itself using a standard Microsoft Edge UA and otherwise looked legitimate, apart from the fact that it was making an absurd number of requests per second. We can also block IPs at a country level if necessary, but we're extremely reluctant to do so unless absolutely necessary as we have customers with global customers. This particular IP was in Canada, which we certainly wouldn't be able to ban.
I'm aware of Apache Rate Limiter mod and have previously implemented it on an older server, but we had issues where it would limit legitimate users unless we set extremely permissive options, which the bots then exploited rendering it worthless. We also use the speed of our hosting servers as a selling point, so I wouldn't want to put anything in place that would slow down browsing for legitimate users.
Any suggestions would be welcome, because I feel I can't be the first person to have found themselves exasperated by this issue, yet I can't find anything particularly useful online.