Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a load balancer using apache: http://httpd.apache.org/docs/2.2/mod/mod%5Fproxy%5Fbalancer.html

The problem is our bandwidth. We're trying to get more, but the ISP has to run new lines and keeps putting us off, so I'd like to throttle down the spiders to conserve bandwidth until we can get more. I tried mod cband, but it won't work on load balanced virtual hosts.

Is there any apache modules that can throttle traffic on load balanced virtual hosts?

share|improve this question
1  
What operating system are you running Apache on? –  PP. Nov 19 '09 at 17:20

1 Answer 1

Not an apache module, but you can try using the "Crawl delay" directive in robots.txt to slow down well behaved spiders.

http://en.wikipedia.org/wiki/Robots%5Fexclusion%5Fstandard#Crawl-delay%5Fdirective

share|improve this answer
    
We've done that, the problem is the ones that don't listen. –  dan Oct 22 '09 at 1:30
1  
If they don't obey robots.txt, then they don't get to index your site until your ISP gets you more bandwidth. Use either a firewall of your choice or Apache to deny access to those IPs. The list of IPs may be rather long, but you should be able to put enough blocks in to hold you over until you can re-allow them. –  Kevin M Nov 19 '09 at 17:23

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.