0

From last couple of weeks I have been experiencing bot attacks on my site. Basically crawlers are running on the site at a high frequency rate resulting in load increase. This results in bandwidth consumption and thus poor user experience for rest of the people. I would like to know how sites like google, amazon and e-bay prevent these things:

  1. How do they differentiate between useful crawlers(google/yahoo/msn) and malicious ones which do not follow robots.txt and other rules?
  2. How can I identify a pattern which can lead to a potential harmful bot in real time by checking the apache access logs?

The threshold values like connections or packets/time/ip can not be set as this will result in poor user experience for the proper customers on the site.

2 Answers 2

1

Here is what I did, and what I recommend:

Create the following script that runs on cron between 1-5 minutes: Read the raw access logs (say, the last 20,000 lines - depending on how active your site is), organize the data by IP/hostname, and then by unix time with attached results. Loop through the results, and identify which IP's have not made any requests to known and required page elements, like images, js, css, etc. From those, count how many page requests were made in one second. I'd characterize 4 page requests or more in a second as excessive and abusive. From there, check against a whitelist of good bots (preg_match against hostname for things like "googlebot.com", etc.), and store all the ip and hostname of the results not white-listed into a database.

Create another script that runs on every page before anything is displayed/processed. Have it validate if the page that is being requested is from an IP that is in your database of banned IP's. If it is, return a 403 with a captcha form prompting the user to submit for reconsideration. If they do, and it validates, have the script unban them. If they aren't banned, store this in a session to avoid repetitive validation for that visitor.

Finally, install monit on your server, and configure it to monitor http once per minute. If the server is unresponsive, it'll restart automatically. This will help to minimize downtime and extreme lag. The script I outlined above will automatically identify scrapers and bad bots, and revoke access. It will also automate reconsideration requests if there are any.

1
0

A solution like Cloudflare or Project HoneyPot should do the trick. Look at this for a few other pointers - https://drupal.stackexchange.com/questions/45678/watchdog-404-page-not-found-entries-from-spambot-guessing-urls/48448#comment50114_48448

Not the answer you're looking for? Browse other questions tagged .