The numbers you are talking about here are well within the range of a single, badly behaved bot. The fact that you are hitting your Apache MaxChildren
limit rather than memory, bandwidth or CPU limits and that they are making valid HTTP requests also leans in favour of the bad bot rather than a deliberate DDoS.
If it is a bot, there will probably be a small range or even a single IP address responsible. When it's going on, grepping through your log files can identify the IP addresses concerned:
tail -5000 access.log | cut -d' ' -f1 | sort | uniq -c | sort -n
Tune the number in the tail to match how busy your sites are.
For simultaneous connections, you can also look at netstat
:
netstat -tan | grep ESTABLISHED | awk '$4 ~ /:80/ {print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
Grep through your access logs for any IP address identified by netstat
and make sure it is not just an enthusiastic user or even your own office IP address.
Once you can tell the difference between someone deliberately trying to take your site down and someone just trying to crawl it as fast as possible, you can take steps to block it.
An example of this is the "Dig; Ext" bot. This bot is used for scraping email addresses from HTML pages. It has no pause between page requests and it runs multiple threads at a time (my guess is 10 based on my access logs). When it comes, it can put an extra 1000 requests per minute in my access logs. It conveniently ads the string "Dig; Ext" to its User-Agent, enabling us to block it easily. A 3 second page load with PHP can be dropped to a few hundred microseconds if Apache returns a 403
based on the User-Agent.
Since these are full TCP connections, the IPs can't be spoofed. Therefore, blocking on IP address should be effective. If the IP ranges are dynamic, ADSL or similar then dropping the packets at your firewall is appropriate. If they are server hosting ranges, it's possible that this is due to incompetence and not malice and it's nice to send back a 403 response as a hint that they might want to make their bot a little better behaved.
The mod_security OWASP core rule set has rules for the bot I mentioned above and many others, along with heuristics that match characteristics of many other bots but no real browsers, such as having no Accept:
headers.
180.76.5.
and all is well.