14

In order to:

  1. Increase security of my website
  2. Reduce bandwidth requirements
  3. Prevent email address harvesting
Improve this question

5 Answers 5

Reset to default
17

No bot that is harvesting emails or testing your site for vulnerabilities will respect your robots.txt. In fact these malicious bots look at the robots.txt to better map your site. If any point you have a Disallow: this will be used to better attack your site. A hacker that is manually looking at your site should spend extra time examining any files/directories that you are attempting to disallow.

Improve this answer
10
  • 3
    Interesting point. I wonder if it makes sense to add a fake page to the Disallow list whose sole purpose is to catch such bots in the act so that they can be automatically blocked.
    – Steven Sudit
    Aug 9, 2010 at 18:28
  • 5
    @Steven Sudit that is not a bad idea. This would be called a honey pot.
    – Rook
    Aug 9, 2010 at 18:42
  • Yes, that's exactly right, although I was thinking more of the standard trick of directory compilers (phone books, etc) of adding a small number of fake entries so as to detect wholesale theft.
    – Steven Sudit
    Aug 9, 2010 at 19:12
  • If you also use that honeypot as a tarpit it will upset the illegit indexers as well. This actually fairly common to do to spammers - leave a honeypot indexable email address that leads to a tarpit email server.
    – Mark Henderson
    Aug 9, 2010 at 21:29
  • @Farseeker I do not see a problem with punishing the ones who break the rules. Although tricking spammers with bad data is an interesting twist.
    – Rook
    Aug 9, 2010 at 22:52
4

robots.txt will not increase security of your website or prevent e-mail address harvesting. robots.txt is a guide for search engines to skip over sections of your website. These will not be indexed and should be used for any sections you do not want to show up in public search engines.

However, this will in no way prevent any other bots from downloading your entire site to increase security or prevent e-mail harvesting. To increase security you need to add authentication and only allow authenticated users beyond the secured sections. To prevent e-mail address harvesting don't put e-mails in plain text (or easily decipherable text) on a website.

Improve this answer
1

robots.txt won't help you with security. Any bot that wants to do something shady will ignore it anyway.

Improve this answer
0

The robots.txt file only serves as a request that bots and spiders leave certain pieces of content alone; it can't actually prevent their access. The "good" bots will respect it, but the "bad" ones (probably the ones that you want to block) will ignore it and proceeed anyway.

Improve this answer
-1

Instead of robots.txt, maybe you need to use CAPTCHA codes.

Improve this answer
2
  • CAPTCHA codes has nothing to do with web crawlers (which is what robots.txt addresses).
    – user48838
    Aug 10, 2010 at 1:39
  • That was an erroneous downvote. The whole point is that a crawler can ignore robots.txt, but CAPTCHA codes will at least slow it down, if not block it outright. Thank you for being wrong.
    – Steven Sudit
    Sep 12, 2010 at 3:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.