Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

From last couple of weeks I have been experiencing bot attacks on my site. Basically crawlers are running on the site at a high frequency rate resulting in load increase. This results in bandwidth consumption and thus poor user experience for rest of the people. I would like to know how sites like google, amazon and e-bay prevent these things:

  1. How do they differentiate between useful crawlers(google/yahoo/msn) and malicious ones which do not follow robots.txt and other rules?
  2. How can I identify a pattern which can lead to a potential harmful bot in real time by checking the apache access logs?

The threshold values like connections or packets/time/ip can not be set as this will result in poor user experience for the proper customers on the site.

share|improve this question

closed as primarily opinion-based by Tom O'Connor Aug 25 '13 at 21:32

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise.If this question can be reworded to fit the rules in the help center, please edit the question.

Here is what I did, and what I recommend:

Create the following script that runs on cron between 1-5 minutes: Read the raw access logs (say, the last 20,000 lines - depending on how active your site is), organize the data by IP/hostname, and then by unix time with attached results. Loop through the results, and identify which IP's have not made any requests to known and required page elements, like images, js, css, etc. From those, count how many page requests were made in one second. I'd characterize 4 page requests or more in a second as excessive and abusive. From there, check against a whitelist of good bots (preg_match against hostname for things like "", etc.), and store all the ip and hostname of the results not white-listed into a database.

Create another script that runs on every page before anything is displayed/processed. Have it validate if the page that is being requested is from an IP that is in your database of banned IP's. If it is, return a 403 with a captcha form prompting the user to submit for reconsideration. If they do, and it validates, have the script unban them. If they aren't banned, store this in a session to avoid repetitive validation for that visitor.

Finally, install monit on your server, and configure it to monitor http once per minute. If the server is unresponsive, it'll restart automatically. This will help to minimize downtime and extreme lag. The script I outlined above will automatically identify scrapers and bad bots, and revoke access. It will also automate reconsideration requests if there are any.

share|improve this answer
Hi @Don, I have reposted the same question here :… – bilkulbekar Feb 15 '12 at 12:03

A solution like Cloudflare or Project HoneyPot should do the trick. Look at this for a few other pointers -

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.