1

I have a bunch of files at www.example.com/A/B/C/NAME (A,B,C change around, NAME is static) and I basically want to add a command in robots.txt so crawlers don't follow any such links that have NAME at the end.

What's the best command to use in robots.txt for this?

1

7 Answers 7

4

it cannot be done. there is no official standard for robots.txt, it's really just a convention that various web-crawlers are trying to respect and correctly interpret.

However Googlebot supports wildcards, so you could have section like this:

User-agent: Googlebot
Disallow: /*NAME

since most web-crawlers won't interpret wildcards correctly and who knows how they interpret it, it's probably safe to isolate this rule just for googlebot but I would assume that by now every large search engine could support it as well as whatever Google does in search becomes de-facto standard.

1
  • Good write up, yahoo(slurp) also supports this exclusion pattern
    – curtisk
    Feb 19, 2010 at 19:39
1
    User-agent: googlebot
    Disallow: /*NAME

    User-Agent: slurp
    Disallow: /*NAME
2
  • Globbing is not allowed in the file so the * does not work. The only reason it works for User-agent is due to the fact that it is handled specifically different in that line.
    – palehorse
    Feb 19, 2010 at 19:22
  • it does in fact work with google and yahoo...updated above, I neglected to define the agent
    – curtisk
    Feb 19, 2010 at 19:32
0

I see you cross-posted this on Stack Overflow, but I'll put my answer here as well.

You cannot glob in the Disallow line unfortunately, so no wildcards. You would need to have a disallow line for each directory you want to exclude.

User-agent: *
Disallow: /A/B/C/NAME/
Disallow: /D/E/F/NAME/

It's unfortunate, but the standards is very simplistic and this is how it needs to be done. Also note you must have the trailing / in your disallow line. Here is a fairly good reference for using robots.txt.

0

To my knowledge there is no pattern matching routine supported by the robots.txt file parsers. In this case you would need to list each of those files with their own Disallow statement.

Keep in mind that listing those files in the robots.txt file will give out a list of those links to anyone who might want to see what you're trying to "hide" from the crawlers, so there may be a security issue if this is sensitive material.

If these links are in HTML served up by your server you can also add a rel="nofollow" to the A tags to those links and it will prevent most crawlers from following the links.

0

As previously mentioned, the robots.txt spec is pretty simple. However, one thing that I've done is create a dynamic script (PHP, Python, whatever) that's simply named "robots.txt" and have it smartly generate the expected, simple structure using the more intelligent logic of the script. You can walk subdirectories, use regular expressions, etc.

You might have to tweak your web server a bit so it executes "robots.txt" as a script rather than just serving up the file contents. Alternatively, you can have a script run via a cron job that regenerates your robots.txt once a night (or however often it needs updating)

0

You cannot glob in the Disallow line unfortunately, so no wildcards. You would need to have a disallow line for each directory you want to exclude.

User-agent: *
Disallow: /A/B/C/NAME/
Disallow: /D/E/F/NAME/

It's unfortunate, but the standards is very simplistic and this is how it needs to be done. Also note you must have the trailing / in your disallow line. Here is a fairly good reference for using robots.txt.

1
  • Yahoo(slurp) and google(googlebot) both support an expanded set of matches, including wildcards, since there's no governing body on robots.txt they decided to roll that out
    – curtisk
    Feb 19, 2010 at 19:35
0

Best documentation I've seen for this is at robotstxt.org.

You must log in to answer this question.