I'm a bit vague on the precise syntax of robots.txt, but what I'm trying to achieve is:

  • Tell all user agents not to crawl certain pages
  • Tell certain user agents not to crawl anything

(basically, some pages with enormous amounts of data should never be crawled; and some voracious but useless search engines, e.g. Cuil, should never crawl anything)

If I do something like this:

User-agent: *
Disallow: /path/page1.aspx
Disallow: /path/page2.aspx
Disallow: /path/page3.aspx

User-agent: twiceler
Disallow: /

..will it flow through as expected, with all user agents matching the first rule and skipping page1, page2 and page3; and twiceler matching the second rule and skipping everything?


It would appear that you have a better understanding than you realize. :)

  • So that all looks good to you? It will use the first rule everything and the second rule for twiceler? – Carson63000 Jun 8 '10 at 6:35

Hmm, depends on the crawler and whether it just goes on first match basis. IE twiceler might see the wildcard entry first and not check any further, so would not see the Disallow: /

  • Would it be recommended to put the wildcard User-agent entry last in the robots.txt file, then? – Carson63000 Jun 11 '10 at 4:48
  • Think og the wildcard as a catchall rule, so I always put it last. Have the specific ones first. – user45348 Jun 11 '10 at 10:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.