14

In order to:

  1. Increase security of my website
  2. Reduce bandwidth requirements
  3. Prevent email address harvesting

5 Answers 5

17

No bot that is harvesting emails or testing your site for vulnerabilities will respect your robots.txt. In fact these malicious bots look at the robots.txt to better map your site. If any point you have a Disallow: this will be used to better attack your site. A hacker that is manually looking at your site should spend extra time examining any files/directories that you are attempting to disallow.

10
  • 3
    Interesting point. I wonder if it makes sense to add a fake page to the Disallow list whose sole purpose is to catch such bots in the act so that they can be automatically blocked. Aug 9, 2010 at 18:28
  • 5
    @Steven Sudit that is not a bad idea. This would be called a honey pot.
    – Rook
    Aug 9, 2010 at 18:42
  • Yes, that's exactly right, although I was thinking more of the standard trick of directory compilers (phone books, etc) of adding a small number of fake entries so as to detect wholesale theft. Aug 9, 2010 at 19:12
  • If you also use that honeypot as a tarpit it will upset the illegit indexers as well. This actually fairly common to do to spammers - leave a honeypot indexable email address that leads to a tarpit email server. Aug 9, 2010 at 21:29
  • @Farseeker I do not see a problem with punishing the ones who break the rules. Although tricking spammers with bad data is an interesting twist.
    – Rook
    Aug 9, 2010 at 22:52
4

robots.txt will not increase security of your website or prevent e-mail address harvesting. robots.txt is a guide for search engines to skip over sections of your website. These will not be indexed and should be used for any sections you do not want to show up in public search engines.

However, this will in no way prevent any other bots from downloading your entire site to increase security or prevent e-mail harvesting. To increase security you need to add authentication and only allow authenticated users beyond the secured sections. To prevent e-mail address harvesting don't put e-mails in plain text (or easily decipherable text) on a website.

1

robots.txt won't help you with security. Any bot that wants to do something shady will ignore it anyway.

0

The robots.txt file only serves as a request that bots and spiders leave certain pieces of content alone; it can't actually prevent their access. The "good" bots will respect it, but the "bad" ones (probably the ones that you want to block) will ignore it and proceeed anyway.

-1

Instead of robots.txt, maybe you need to use CAPTCHA codes.

2
  • CAPTCHA codes has nothing to do with web crawlers (which is what robots.txt addresses).
    – user48838
    Aug 10, 2010 at 1:39
  • That was an erroneous downvote. The whole point is that a crawler can ignore robots.txt, but CAPTCHA codes will at least slow it down, if not block it outright. Thank you for being wrong. Sep 12, 2010 at 3:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .