we run a commerical website with a relavitely few number of customer at any one time ~30 users .

Frequently a webcrawler such as google bot, bing bot, and 80legs will bring our site to a grinding halt. Altering robots.txt does not have an immediate effect. Is their anyway to stop 80legs instantly ? It is using different ip's.


From http://wiki.80legs.com/w/page/1114616/FAQ#HowdoInbspslowdown008whencrawlingmysite

How do I slow down 008 when crawling my site?

The easiest solution is to contact us and let us know at what rate you would like us to throttle crawls on your site (in terms of requests/second).

Of course you can also check for the User-Agent and rate limit and/or block these requests completely.

  • yeah thanks but none of that is particularly fast, I have altered robots.tx tbut that is only picked up by them every 3hrs - we are getting the equivalent ddos attack – NimChimpsky Jun 13 '12 at 15:33
  • 1
    Blocking based on User-Agent would be instant – faker Jun 13 '12 at 16:04
  • only if they pick it up instantly and actually obey the rules, which they don't apparently. – NimChimpsky Jun 13 '12 at 16:05
  • No, you tell your webserver to check for the User-Agent and block these requests. The only way to get around this for them is to change the User-Agent - this is possible, but a commercial provider will probably not do this – faker Jun 13 '12 at 16:12
  • oh, how would I tell tomcat to block a user agent without using robots.txt, which relies on co-operation of the bot ? – NimChimpsky Jun 13 '12 at 16:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.