Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

In order to:

  1. Increase security of my website
  2. Reduce bandwidth requirements
  3. Prevent email address harvesting
share|improve this question

migrated from stackoverflow.com Aug 9 '10 at 19:43

This question came from our site for professional and enthusiast programmers.

add comment

5 Answers

up vote 16 down vote accepted

No bot that is harvesting emails or testing your site for vulnerabilities will respect your robots.txt. In fact these malicious bots look at the robots.txt to better map your site. If any point you have a Disallow: this will be used to better attack your site. A hacker that is manually looking at your site should spend extra time examining any files/directories that you are attempting to disallow.

share|improve this answer
3  
Interesting point. I wonder if it makes sense to add a fake page to the Disallow list whose sole purpose is to catch such bots in the act so that they can be automatically blocked. –  Steven Sudit Aug 9 '10 at 18:28
5  
@Steven Sudit that is not a bad idea. This would be called a honey pot. –  Rook Aug 9 '10 at 18:42
    
Yes, that's exactly right, although I was thinking more of the standard trick of directory compilers (phone books, etc) of adding a small number of fake entries so as to detect wholesale theft. –  Steven Sudit Aug 9 '10 at 19:12
    
If you also use that honeypot as a tarpit it will upset the illegit indexers as well. This actually fairly common to do to spammers - leave a honeypot indexable email address that leads to a tarpit email server. –  Mark Henderson Aug 9 '10 at 21:29
    
@Farseeker I do not see a problem with punishing the ones who break the rules. Although tricking spammers with bad data is an interesting twist. –  Rook Aug 9 '10 at 22:52
show 5 more comments

robots.txt will not increase security of your website or prevent e-mail address harvesting. robots.txt is a guide for search engines to skip over sections of your website. These will not be indexed and should be used for any sections you do not want to show up in public search engines.

However, this will in no way prevent any other bots from downloading your entire site to increase security or prevent e-mail harvesting. To increase security you need to add authentication and only allow authenticated users beyond the secured sections. To prevent e-mail address harvesting don't put e-mails in plain text (or easily decipherable text) on a website.

share|improve this answer
add comment

robots.txt won't help you with security. Any bot that wants to do something shady will ignore it anyway.

share|improve this answer
add comment

The robots.txt file only serves as a request that bots and spiders leave certain pieces of content alone; it can't actually prevent their access. The "good" bots will respect it, but the "bad" ones (probably the ones that you want to block) will ignore it and proceeed anyway.

share|improve this answer
add comment

Instead of robots.txt, maybe you need to use CAPTCHA codes.

share|improve this answer
    
CAPTCHA codes has nothing to do with web crawlers (which is what robots.txt addresses). –  user48838 Aug 10 '10 at 1:39
    
That was an erroneous downvote. The whole point is that a crawler can ignore robots.txt, but CAPTCHA codes will at least slow it down, if not block it outright. Thank you for being wrong. –  Steven Sudit Sep 12 '10 at 3:52
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.