0

Today MySQL was killed off by the oom-killer on one of our shared web servers, which caused an outage for our customers. When I investigated what happened I found there were over 1,300 requests in the space of 20 seconds from one IP address. They hit every vhost on the server which meant PHP-FPM had to be spun up for every single site, which in turn led to the OOM situation. I'm going to be adjusting the oom-killer to avoid having it kill MySQL in the future, but that's not what this question is about.

What I want to know is whether there is a sensible way to detect this situation in the future. No bot needs to be making >66 requests per second, especially not across multiple vhosts. It feels like it ought to be possible to detect large, sudden spikes in traffic going to multiple vhosts, and take appropriate action to stop them.

We already implement fail2ban but this doesn't feel like it truly falls under its purview, and I certainly can't see a sane way of implementing a check using that tool which wouldn't run the risk of blocking legitimate spiders that cross vhosts without causing issues (e.g. Google Bot and Bing Bot). We also use Varnish in front of the web server to ease the burden on Apache, and from there we block known bad bot UAs and other tells from bots. But this bot identified itself using a standard Microsoft Edge UA and otherwise looked legitimate, apart from the fact that it was making an absurd number of requests per second. We can also block IPs at a country level if necessary, but we're extremely reluctant to do so unless absolutely necessary as we have customers with global customers. This particular IP was in Canada, which we certainly wouldn't be able to ban.

I'm aware of Apache Rate Limiter mod and have previously implemented it on an older server, but we had issues where it would limit legitimate users unless we set extremely permissive options, which the bots then exploited rendering it worthless. We also use the speed of our hosting servers as a selling point, so I wouldn't want to put anything in place that would slow down browsing for legitimate users.

Any suggestions would be welcome, because I feel I can't be the first person to have found themselves exasperated by this issue, yet I can't find anything particularly useful online.

  • 1
    Per IP rate limiting ? – Iain May 15 at 14:23
  • 1
    Rate-limits, of course. User-Agents black-listing. Fail2ban, matching "suspicious" queries in your webserver logs (whoever's accessing /phpmyadmin, /wp-admin, ...). To avoid OOMs, maybe add some swap space. Although I'ld first recommend not to collocate database with webapps. – SYN May 15 at 15:52
  • We already do all of these things (including the swap) except for the rate limiting for the reasons I mentioned in the original question. Unless you know of a way of doing this that is less prone to causing false positive issues than mod_ratelimit. We plan to move the database to its own dedicated server at some time in the future as I fully agree it's suboptimal having it on the same system. – Pete Michaels May 15 at 17:05
  • What is your php-fpm process manager configuration? Why don't you have enough memory to start the maximum number of child processes? – John Mahowald May 16 at 11:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.