We have a simple load balancer set up on Rackspace to 3 web server nodes. After reviewing our traffic and expenses, the largest bandwidth hog is Google Bot. Since on Rackspace we pay for bandwidth by the byte, we'd like to direct all traffic from GoogleBot to another host (MediaTemple) with unlimited bandwidth.

We think this would cut our hosting bill several thousand dollars a month.

Is this possible? Advisable?

  • Doh, I'm primarily a SO user and keep my rate up there but didn't realize I had neglected this. Fixed. Thanks. – Ryan Nov 13 '12 at 22:28

It's possible but will complicate your infrastructure and you'd need a global load balancer hosted somewhere (AWS maybe) infront of rackspace. Also you need to be careful here, if Googlebot detects that it's not your live site then this may hurt your ranking.

Other options, limit googlebot while your monitoring rankings or talk to Rackspace see what they can offer generally in my experience they have some excellent people that can advise here (on and off the record).

Think the best option is to identify the page content that google is crawling and cache that on a CDN, the bandwidth will be offloaded elsewhere (should be cheaper) and as a bonus you get a more scalable solution as well.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.