2

We have a situation where bots comprise 40% of page requests for a slow-performing web application.

Short Term Relief

As immediate relief, I thought of doing the following:

  • Setting up another instance of the application pointing to a slave copy of the database. Call this "Server B" , whereas the main application is "Server A")
  • forwarding all bot-traffic to "Server B", forward all other requests to the main application "Server A"

Separating out, by server, "bot traffic" from "user traffic" would give the added benefit of more accurate server side analysis (i.e. knowing that all traffic is from "real users" and not from bots). Additionally we could use a slower server for "bot traffic" and dedicate more resources to "bona-fide customers",

Context

  • currently using apache 2.2 with mod_proxy front end.
  • Back end is jboss/tomcat
  • We have a manually-maintained list of bot headers (e.g. baidu,bingbot,googlebot,msnbot, etc.)

Question:

  • Can apache (or mod_proxy) forward requests based on the headers (i.e. forward all requests with baidu,googlebot,msnbot etc in the header to ServerB, all others to Server A)

Please weigh in with any other suggestions

Thanks in advance

2 Answers 2

3

At risk of getting downvoted for not actually answering your question:

  1. You can slow Googlebot down by using a setting in Google Webmaster Tools.

  2. Most of the major crawlers support the Crawl-delay directive in robots.txt to slow them down. Yahoo's bot can be particularly aggressive.

  3. Ban the likes of Baidu completely in robots.txt. They're not doing you any good.

  4. Think of aggressive bots as DOS attacks, and limit the frequency of requests from each IP using either IPTABLES or the mod_security plugin for Apache.

1

A couple of suggestions:

  • Look at using the User-Agent to route the bots (or something similar) to a static page so your application is not spending cycles responding to the bots. See mod rewrite guide which has some examples
  • Is using robots.txt an option? This might be the easiest option to alleviate load on your server. Adding another server could be 'expensive' (in time and maintenance), and would not be my first choice.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .