Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a bunch of files at www.example.com/A/B/C/NAME (A,B,C change around, NAME is static) and I basically want to add a command in robots.txt so crawlers don't follow any such links that have NAME at the end.

What's the best command to use in robots.txt for this?

share|improve this question
    
duplicate of serverfault.com/questions/114754/robots-txt-command –  pauska Feb 19 '10 at 19:58
add comment

7 Answers

it cannot be done. there is no official standard for robots.txt, it's really just a convention that various web-crawlers are trying to respect and correctly interpret.

However Googlebot supports wildcards, so you could have section like this:

User-agent: Googlebot
Disallow: /*NAME

since most web-crawlers won't interpret wildcards correctly and who knows how they interpret it, it's probably safe to isolate this rule just for googlebot but I would assume that by now every large search engine could support it as well as whatever Google does in search becomes de-facto standard.

share|improve this answer
    
Good write up, yahoo(slurp) also supports this exclusion pattern –  curtisk Feb 19 '10 at 19:39
add comment
    User-agent: googlebot
    Disallow: /*NAME

    User-Agent: slurp
    Disallow: /*NAME
share|improve this answer
    
Globbing is not allowed in the file so the * does not work. The only reason it works for User-agent is due to the fact that it is handled specifically different in that line. –  palehorse Feb 19 '10 at 19:22
    
it does in fact work with google and yahoo...updated above, I neglected to define the agent –  curtisk Feb 19 '10 at 19:32
add comment

I see you cross-posted this on Stack Overflow, but I'll put my answer here as well.

You cannot glob in the Disallow line unfortunately, so no wildcards. You would need to have a disallow line for each directory you want to exclude.

User-agent: *
Disallow: /A/B/C/NAME/
Disallow: /D/E/F/NAME/

It's unfortunate, but the standards is very simplistic and this is how it needs to be done. Also note you must have the trailing / in your disallow line. Here is a fairly good reference for using robots.txt.

share|improve this answer
add comment

To my knowledge there is no pattern matching routine supported by the robots.txt file parsers. In this case you would need to list each of those files with their own Disallow statement.

Keep in mind that listing those files in the robots.txt file will give out a list of those links to anyone who might want to see what you're trying to "hide" from the crawlers, so there may be a security issue if this is sensitive material.

If these links are in HTML served up by your server you can also add a rel="nofollow" to the A tags to those links and it will prevent most crawlers from following the links.

share|improve this answer
add comment

As previously mentioned, the robots.txt spec is pretty simple. However, one thing that I've done is create a dynamic script (PHP, Python, whatever) that's simply named "robots.txt" and have it smartly generate the expected, simple structure using the more intelligent logic of the script. You can walk subdirectories, use regular expressions, etc.

You might have to tweak your web server a bit so it executes "robots.txt" as a script rather than just serving up the file contents. Alternatively, you can have a script run via a cron job that regenerates your robots.txt once a night (or however often it needs updating)

share|improve this answer
add comment

You cannot glob in the Disallow line unfortunately, so no wildcards. You would need to have a disallow line for each directory you want to exclude.

User-agent: *
Disallow: /A/B/C/NAME/
Disallow: /D/E/F/NAME/

It's unfortunate, but the standards is very simplistic and this is how it needs to be done. Also note you must have the trailing / in your disallow line. Here is a fairly good reference for using robots.txt.

share|improve this answer
    
Yahoo(slurp) and google(googlebot) both support an expanded set of matches, including wildcards, since there's no governing body on robots.txt they decided to roll that out –  curtisk Feb 19 '10 at 19:35
add comment

Best documentation I've seen for this is at robotstxt.org.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.