Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am denying indexing to a folder called pdf via robots.txt. However, I do direct link to a few files that exist in that directory.

Will search engines such as Google index those files, or ignore them because they reside in the pdf folder?

share|improve this question
up vote 1 down vote accepted

Short answer: No.

Crawlers are disallowed from indexing anything under the URL prefix you put in robots.txt.

Longer answer: It depends.

The Allow keyword is not part of the standard but some robots will follow it. You can use this to Allow a particular URL and Disallow the entire subtree that contains that URL. Most bots work on a first-match-wins basis. Google and Bing work on a longest-string-wins basis regardless of the order of the Allow and Disallow lines.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.