Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a load balancer using apache:

The problem is our bandwidth. We're trying to get more, but the ISP has to run new lines and keeps putting us off, so I'd like to throttle down the spiders to conserve bandwidth until we can get more. I tried mod cband, but it won't work on load balanced virtual hosts.

Is there any apache modules that can throttle traffic on load balanced virtual hosts?

share|improve this question
What operating system are you running Apache on? – PP. Nov 19 '09 at 17:20

Not an apache module, but you can try using the "Crawl delay" directive in robots.txt to slow down well behaved spiders.

share|improve this answer
We've done that, the problem is the ones that don't listen. – dan Oct 22 '09 at 1:30
If they don't obey robots.txt, then they don't get to index your site until your ISP gets you more bandwidth. Use either a firewall of your choice or Apache to deny access to those IPs. The list of IPs may be rather long, but you should be able to put enough blocks in to hold you over until you can re-allow them. – Kevin M Nov 19 '09 at 17:23

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.