A webcrawler has bought our site down twice, it ignores our robots.txt and we have had no reply from their customer services or support using both e-mail and twitter.
I have had to create url redirect based on their user agent string, I have redirected all their requests back to their own public website. Is this the right thing to do ?
Edit How do I return a 40? error code based on user agent string using tomcat/tuckey ? I can't use ip addresses as the bot uses many (its grid based apparently).
This is partly due to our website being an old and creaky legacy system, which is a question for a certian other site. But google bot and bing bot do not knock us over, and out normal business traffic is just fine. A significant investment/development to handle one bot is not sensible. We are hosted on a windows server.