13
votes
5answers
1k views

Which bots and spiders should I block in robots.txt?

In order to: Increase security of my website Reduce bandwidth requirements Prevent email address harvesting