Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have a simple load balancer set up on Rackspace to 3 web server nodes. After reviewing our traffic and expenses, the largest bandwidth hog is Google Bot. Since on Rackspace we pay for bandwidth by the byte, we'd like to direct all traffic from GoogleBot to another host (MediaTemple) with unlimited bandwidth.

We think this would cut our hosting bill several thousand dollars a month.

Is this possible? Advisable?

share|improve this question

migrated from superuser.com Nov 13 '12 at 19:36

This question came from our site for computer enthusiasts and power users.

You might want to go and accept some answers on previous questions, as a 0% accept rate possibly won't get you much answers here. –  Frederik Nielsen Nov 13 '12 at 19:41
Doh, I'm primarily a SO user and keep my rate up there but didn't realize I had neglected this. Fixed. Thanks. –  Ryan Nov 13 '12 at 22:28

1 Answer 1

It's possible but will complicate your infrastructure and you'd need a global load balancer hosted somewhere (AWS maybe) infront of rackspace. Also you need to be careful here, if Googlebot detects that it's not your live site then this may hurt your ranking.

Other options, limit googlebot while your monitoring rankings or talk to Rackspace see what they can offer generally in my experience they have some excellent people that can advise here (on and off the record).

Think the best option is to identify the page content that google is crawling and cache that on a CDN, the bandwidth will be offloaded elsewhere (should be cheaper) and as a bonus you get a more scalable solution as well.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.