Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Will the following robots.txt syntax correctly block all pages on the site that end in "_.php"? I don't want to accidentally block other pages.

User-Agent: *    
Disallow: /*_.php

Also, am I allowed to have both "Allow: /" and "Disallow:" commands in the same robots.txt file? Thanks!

share|improve this question
    
The first line should begin with "User-agent" and not "User-Agent"; it is very reasonable to expect some/most crawlers to parse your robots.txt file with case-sensitivity, so a "User-Agent" field could be disregarded entirely. –  Skyhawk Jul 23 '10 at 22:32

2 Answers 2

up vote 2 down vote accepted

If you want certain files (but not others) excluded, you must group them into directories, e.g.:

User-agent: *
Disallow: /cgi-bin/

Per robotstxt.org, asterisks are not supported in the "Disallow" field:

Note also that globbing and regular expression are not supported in either the User-agent or Disallow lines. The '*' in the User-agent field is a special value meaning "any robot". Specifically, you cannot have lines like "User-agent: bot", "Disallow: /tmp/*" or "Disallow: *.gif".

Additionally, there is no such thing as an "Allow" field. Everything is allowed by default, and specific items are disallowed by exception.

References:

share|improve this answer
    
I'm developing some pages that I don't want crawled yet, and they exist on the server in the form of filename_.php instead of filename.php. How would you propose I block those files from being crawled? A meta tag in each one's <head>? –  bccarlso Jul 23 '10 at 22:32
    
Standard practice would be to build separate development and test environments. In a full-on professional operation, nothing is moved to an internet-facing production server until it is ready for prime time. Options include installing XAMPP on your development workstation and using it as a development environment (localhost), or creating a subdomain on your hosted site (test.yourdomain.com), or perhaps even just uploading a copy of the site to a subfolder (yourdomain.com/test). You can use .htpasswd to enable minimal security for an internet-facing test site. –  Skyhawk Jul 23 '10 at 22:41
    
Another thought: it's also important to note that robots.txt is a voluntary system to "suggest" that certain pages should not be crawled. It may stop good crawlers, but it certainly will not stop evil ones. –  Skyhawk Jul 23 '10 at 22:45
    
Thanks Miles. Yeah, I had heard that robots.txt is not a hard rule, and also know I should have developed in an offline or protected online environment. It was one of those "was just going to be one page but turned into the whole site" redesigns/redevelopments, and it was my fault for not moving it offline. Thanks for the help. –  bccarlso Jul 26 '10 at 15:10

Miles' answer covers the standards. The most famous crawler, Googlebot, extends the standards and does understand Allow as well as (limited) pattern matching.

I find Google's webmaster tools quite helpful. They have a whole tool devoted to just helping you build a correct robots.txt. You do need to have the pages (or at least stub test pages) uploaded before you can run a "robots.txt test", though.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.