A webcrawler has bought our site down twice. It ignores our robots.txt and we have had no reply from their customer services or support using both e-mail and twitter.
I have had to create url redirect based on their user agent string, I have redirected all their requests back to their own public website. Is this the right thing to do?
Edit How do I return a 40? error code based on user agent string using tomcat/tuckey? (Our site is hosted on a Windows server, if that matters.) I can't use IP addresses as the bot uses many (it's grid based apparently).
This is partly due to our website being an old and creaky legacy system, but Google's crawler and Bing's crawler do not knock us over, and our normal business traffic is just fine. A significant investment/development to handle one bot is not sensible.