Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Yahoo's robots.txt contains:

User-agent: *
Disallow: /p/
Disallow: /r/
Disallow: /*?

What does the last line mean? ("Disallow: /*?")

share|improve this question

migrated from stackoverflow.com May 7 '10 at 6:20

This question came from our site for professional and enthusiast programmers.

2 Answers 2

If it was a Perl regular expression:

*?     Match 0 or more times, not greedily

http://perldoc.perl.org/perlre.html

However robots.txt follows a really basic grammar, as such,

To match a sequence of characters, use an asterisk (*). For instance, to block access to all subdirectories that begin with private:

User-agent: Googlebot
Disallow: /private*/

To block access to all URLs that include a question mark (?) (more specifically, any URL that begins with your domain name, followed by any string, followed by a question mark, followed by any string):

User-agent: Googlebot
Disallow: /*?

To specify matching the end of a URL, use $. For instance, to block any URLs that end with .xls:

User-agent: Googlebot 
Disallow: /*.xls$

You can use this pattern matching in combination with the Allow directive. For instance, if a ? indicates a session ID, you may want to exclude all URLs that contain them to ensure Googlebot doesn't crawl duplicate pages. But URLs that end with a ? may be the version of the page that you do want included. For this situation, you can set your robots.txt file as follows:

User-agent: *
Allow: /*?$
Disallow: /*?

The Disallow: / *? directive will block any URL that includes a ? (more specifically, it will block any URL that begins with your domain name, followed by any string, followed by a question mark, followed by any string).

The Allow: /*?$ directive will allow any URL that ends in a ? (more specifically, it will allow any URL that begins with your domain name, followed by a string, followed by a ?, with no characters after the ?).

So basically any kind of query or search on Yahoo! is prohibited by a robot.

The expression support is confusingly not listed in the RFC, http://www.robotstxt.org/norobots-rfc.txt

The best description is provided by Google, http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

share|improve this answer
    
so we dont get the link from yahoo –  hussain May 6 '10 at 6:50
    
Every Yahoo! domain has a different robots.txt, check out Yahoo! from the homepage almost every link goes to a different domain with different restrictions. –  Steve-o May 6 '10 at 6:57
    
It is very important to understand that many/most crawlers do not understand wildcards in robots.txt files, because that is not part of the specification. According to robotstxt.org, the only place where an asterisk is explicitly allowed is in the "User-agent" field. –  Skyhawk Jul 23 '10 at 22:50

The * makes it a wildcard. So, uri that ends with a ? would be restricted.

share|improve this answer
    
That would require "/*?$". –  Steve-o May 6 '10 at 6:47

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.