I am denying indexing to a folder called pdf via robots.txt. However, I do direct link to a few files that exist in that directory.

Will search engines such as Google index those files, or ignore them because they reside in the pdf folder?


Short answer: No.

Crawlers are disallowed from indexing anything under the URL prefix you put in robots.txt.

Longer answer: It depends.

The Allow keyword is not part of the standard but some robots will follow it. You can use this to Allow a particular URL and Disallow the entire subtree that contains that URL. Most bots work on a first-match-wins basis. Google and Bing work on a longest-string-wins basis regardless of the order of the Allow and Disallow lines.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.