Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

we run a commerical website with a relavitely few number of customer at any one time ~30 users .

Frequently a webcrawler such as google bot, bing bot, and 80legs will bring our site to a grinding halt. Altering robots.txt does not have an immediate effect. Is their anyway to stop 80legs instantly ? It is using different ip's.

share|improve this question

1 Answer 1

up vote 1 down vote accepted

From http://wiki.80legs.com/w/page/1114616/FAQ#HowdoInbspslowdown008whencrawlingmysite

How do I slow down 008 when crawling my site?

The easiest solution is to contact us and let us know at what rate you would like us to throttle crawls on your site (in terms of requests/second).

Of course you can also check for the User-Agent and rate limit and/or block these requests completely.

share|improve this answer
    
yeah thanks but none of that is particularly fast, I have altered robots.tx tbut that is only picked up by them every 3hrs - we are getting the equivalent ddos attack –  NimChimpsky Jun 13 '12 at 15:33
1  
Blocking based on User-Agent would be instant –  faker Jun 13 '12 at 16:04
    
only if they pick it up instantly and actually obey the rules, which they don't apparently. –  NimChimpsky Jun 13 '12 at 16:05
    
No, you tell your webserver to check for the User-Agent and block these requests. The only way to get around this for them is to change the User-Agent - this is possible, but a commercial provider will probably not do this –  faker Jun 13 '12 at 16:12
    
oh, how would I tell tomcat to block a user agent without using robots.txt, which relies on co-operation of the bot ? –  NimChimpsky Jun 13 '12 at 16:13

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.