Here is what I did, and what I recommend:
Create the following script that runs on cron between 1-5 minutes: Read the raw access logs (say, the last 20,000 lines - depending on how active your site is), organize the data by IP/hostname, and then by unix time with attached results. Loop through the results, and identify which IP's have not made any requests to known and required page elements, like images, js, css, etc. From those, count how many page requests were made in one second. I'd characterize 4 page requests or more in a second as excessive and abusive. From there, check against a whitelist of good bots (preg_match against hostname for things like "googlebot.com", etc.), and store all the ip and hostname of the results not white-listed into a database.
Create another script that runs on every page before anything is displayed/processed. Have it validate if the page that is being requested is from an IP that is in your database of banned IP's. If it is, return a 403 with a captcha form prompting the user to submit for reconsideration. If they do, and it validates, have the script unban them. If they aren't banned, store this in a session to avoid repetitive validation for that visitor.
Finally, install monit on your server, and configure it to monitor http once per minute. If the server is unresponsive, it'll restart automatically. This will help to minimize downtime and extreme lag. The script I outlined above will automatically identify scrapers and bad bots, and revoke access. It will also automate reconsideration requests if there are any.