Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm a bit vague on the precise syntax of robots.txt, but what I'm trying to achieve is:

  • Tell all user agents not to crawl certain pages
  • Tell certain user agents not to crawl anything

(basically, some pages with enormous amounts of data should never be crawled; and some voracious but useless search engines, e.g. Cuil, should never crawl anything)

If I do something like this:

User-agent: *
Disallow: /path/page1.aspx
Disallow: /path/page2.aspx
Disallow: /path/page3.aspx

User-agent: twiceler
Disallow: /

..will it flow through as expected, with all user agents matching the first rule and skipping page1, page2 and page3; and twiceler matching the second rule and skipping everything?

share|improve this question

2 Answers 2

It would appear that you have a better understanding than you realize. :)

share|improve this answer
    
So that all looks good to you? It will use the first rule everything and the second rule for twiceler? –  Carson63000 Jun 8 '10 at 6:35

Hmm, depends on the crawler and whether it just goes on first match basis. IE twiceler might see the wildcard entry first and not check any further, so would not see the Disallow: /

share|improve this answer
    
Would it be recommended to put the wildcard User-agent entry last in the robots.txt file, then? –  Carson63000 Jun 11 '10 at 4:48
    
Think og the wildcard as a catchall rule, so I always put it last. Have the specific ones first. –  user45348 Jun 11 '10 at 10:55

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.