we run a commerical website with a relavitely few number of customer at any one time ~30 users .
Frequently a webcrawler such as google bot, bing bot, and 80legs will bring our site to a grinding halt. Altering robots.txt does not have an immediate effect. Is their anyway to stop 80legs instantly ? It is using different ip's.