I have this robots.txt:

User-Agent: *
Disallow: /files/

User-Agent: ia_archiver
Allow: /

User-agent: Googlebot 

User-agent: googlebot-image 

User-agent: googlebot-mobile 

I am finding that PDF files in the /files/ directory are being indexed by Google.

Should I move the first entry to the bottom?

In working with Google's webmaster tools. I moved the /files/ disallow to the bottom and ran a test on one PDF file in the files directory and it returned Success.

How can I fix this issue? We do not want anything in this directory being indexed.


Even if I remove everything except the first clause,

User-Agent: *
Disallow: /files/

Google still is able to see PDFs in the /files/ directory, what am I doing wrong here?

In Bing's webmaster tools, it shows as blocked but Google's still shows Success.

  • 3
    How long are you waiting after changing the robots.txt before analysing your logs? And which Google bot is retrieving the PDFs? – Ladadadada May 6 '14 at 16:43

Edit: re-read the standard. a robot will use the first matching name token, or fall-back to *. For each bot you want to deny access to /files/, you'll need to add a matching disallow:

User-agent: *
Disallow: /files/

User-agent: Googlebot 
Disallow: /files/

http://www.robotstxt.org/ is a great resource, if you haven't seen it.

  • * should work for all bots that adhere to the standards. – MB34 May 7 '14 at 18:29
  • 1
    thats true, but the standard also says a bot only has to use the first, closest matching rule. so if googlebot finds its token, it will only process that and not bother with the *. google's faq confirms this behavior: support.google.com/webmasters/answer/156449 under 'create a robots.txt file'. i too used to think it 'stacked', but re-read it the other day and apparently not.... – pete May 7 '14 at 18:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.