Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We have a simple load balancer set up on Rackspace to 3 web server nodes. After reviewing our traffic and expenses, the largest bandwidth hog is Google Bot. Since on Rackspace we pay for bandwidth by the byte, we'd like to direct all traffic from GoogleBot to another host (MediaTemple) with unlimited bandwidth.

We think this would cut our hosting bill several thousand dollars a month.

Is this possible? Advisable?

share|improve this question

migrated from Nov 13 '12 at 19:36

This question came from our site for computer enthusiasts and power users.

Doh, I'm primarily a SO user and keep my rate up there but didn't realize I had neglected this. Fixed. Thanks. – Ryan Nov 13 '12 at 22:28

It's possible but will complicate your infrastructure and you'd need a global load balancer hosted somewhere (AWS maybe) infront of rackspace. Also you need to be careful here, if Googlebot detects that it's not your live site then this may hurt your ranking.

Other options, limit googlebot while your monitoring rankings or talk to Rackspace see what they can offer generally in my experience they have some excellent people that can advise here (on and off the record).

Think the best option is to identify the page content that google is crawling and cache that on a CDN, the bandwidth will be offloaded elsewhere (should be cheaper) and as a bonus you get a more scalable solution as well.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.