The lanl.arxiv.org math and scientific preprint service (formerly known as xxx.lanl.gov) has a strict policy against bots that ignore its robots.txt, Robots Beware. On that page, the have a link labelled with "Click here to initiate automated 'seek-and-destroy' against your site", which is forbidden by their robots.txt but presumably badly behaved robots will follow it, and reap the consequences. The question, what are the actual consequences? I have never had the guts to actually click on that link to see what it does. What can they be doing that is both effective and legal?
Tell me more
×
Server Fault is a question and answer site for
professional system and network administrators. It's 100% free, no registration required.
|
|
|||||||
|
So... it's a page that would waste 10 minutes of a very naive bot's time. Probably useless for combating malicious bots, but might save some bandwidth when faced with a badly-written site-scraper. |
|||
|
|
|
No consequences other than spinning for a bit. Most browsers (and probably their server) will just timeout after a bit. They probably just cause more harm to themselves with this than the bots. |
|||
|
|
