A webcrawler has bought our site down twice. It ignores our robots.txt and we have had no reply from their customer services or support using both e-mail and twitter.
I have had to create url redirect based on their user agent string, I have redirected all their requests back to their own public website. Is this the right thing to do?
Edit How do I return a 40? error code based on user agent string using tomcat/tuckey? (Our site is hosted on a Windows server, if that matters.) I can't use IP addresses as the bot uses many (it's grid based apparently).
This is partly due to our website being an old and creaky legacy system, but Google's crawler and Bing's crawler do not knock us over, and our normal business traffic is just fine. A significant investment/development to handle one bot is not sensible.
401
,403
or429
. Or maybe we can get a new status code? Something like437 - Bad Bot
or so... – Bobby Jun 15 '12 at 8:41iptables
. If it isn't useful, 403 or an outright block with iptables is the best bet. Bots don't follow redirects in the same way that browsers do so giving them 1000 redirects to their own home page only results in the bot requesting that page once. (Or whatever firewall you use since I see you're on Windows.) – Ladadadada Jun 15 '12 at 9:35