We have a situation where bots comprise 40% of page requests for a slow-performing web application.
Short Term Relief
As immediate relief, I thought of doing the following:
- Setting up another instance of the application pointing to a slave copy of the database. Call this "Server B" , whereas the main application is "Server A")
- forwarding all bot-traffic to "Server B", forward all other requests to the main application "Server A"
Separating out, by server, "bot traffic" from "user traffic" would give the added benefit of more accurate server side analysis (i.e. knowing that all traffic is from "real users" and not from bots). Additionally we could use a slower server for "bot traffic" and dedicate more resources to "bona-fide customers",
- currently using apache 2.2 with mod_proxy front end.
- Back end is jboss/tomcat
- We have a manually-maintained list of bot headers (e.g. baidu,bingbot,googlebot,msnbot, etc.)
- Can apache (or mod_proxy) forward requests based on the headers (i.e. forward all requests with baidu,googlebot,msnbot etc in the header to ServerB, all others to Server A)
Please weigh in with any other suggestions
Thanks in advance