I have an odd problem-- on a high traffic website (millions of visitors a month), every day we get about 20 or so situations where one host begins incessantly requesting the same page, over and over-- multiple times per second, for any length of time from a few minutes to all day.
The log hits have the following characteristic:
- They start 'normal'-- first page load actually accesses all the page's resources, as well as the .php
- Then the host begins requesting JUST the php page, without the resources incessantly, usually one per second (but sometimes faster and sometimes a few seconds slower)
- The remote browser is always Firefox 3.5.x
- Subsequent hits have no referrer, even though the first page request had one
We are at wit's end with what to do with this. A simple DoS filter is not appropriate-- we have that and the threshold to trigger it is much higher than a single page request (without related images, css, etc.) a second.
The stack is LAMP, Redhat install, PHP 5.2, Apache 2.2.3, with an NGINX box operating as a software load balancer.
This is crushing our site-- please help! In the absence of good ideas, we are going to resort to writing a dummy filter that stores a key of IP+URI in memcached, and increments each page request. Once it crosses a certain threshold in a certain period of time, we'll 403 further requests. I do NOT think this is the appropriate place in the networking stack to handle this issue, however.
Thank you for anything you can contribute!