In order to:
- Increase security of my website
- Reduce bandwidth requirements
- Prevent email address harvesting
Stack Exchange network consists of 181 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.Visit Stack Exchange
Server Fault is a question and answer site for system and network administrators. It only takes a minute to sign up.Sign up to join this community
In order to:
No bot that is harvesting emails or testing your site for vulnerabilities will respect your robots.txt. In fact these malicious bots look at the robots.txt to better map your site. If any point you have a
Disallow: this will be used to better attack your site. A hacker that is manually looking at your site should spend extra time examining any files/directories that you are attempting to disallow.
robots.txt will not increase security of your website or prevent e-mail address harvesting. robots.txt is a guide for search engines to skip over sections of your website. These will not be indexed and should be used for any sections you do not want to show up in public search engines.
However, this will in no way prevent any other bots from downloading your entire site to increase security or prevent e-mail harvesting. To increase security you need to add authentication and only allow authenticated users beyond the secured sections. To prevent e-mail address harvesting don't put e-mails in plain text (or easily decipherable text) on a website.
robots.txt won't help you with security. Any bot that wants to do something shady will ignore it anyway.
The robots.txt file only serves as a request that bots and spiders leave certain pieces of content alone; it can't actually prevent their access. The "good" bots will respect it, but the "bad" ones (probably the ones that you want to block) will ignore it and proceeed anyway.
Instead of robots.txt, maybe you need to use CAPTCHA codes.