Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Yahoo AdCrawler is re-trying some URLs repeatedly. The URLs are being given a 302 response code, so I suppose Yahoo should come back and try again "later", but "later" in my book doesn't mean that 7 specific URLs should be hit 3,000 times a day each.

The URLs are of the style:


The Query part may well be something our Client has added, but I mention it in case it rings any bells

We've banned AdCrawler in Robots.txt for now.

User-agent: Yahoo!-AdCrawler
Disallow: /

If you have any thoughts, similar experiences, or ideas I'd appreciate them. Thanks.

share|improve this question
up vote 0 down vote accepted

AdCrawler is NOT the same as Slurp, which is what Yahoo uses to build their index. AdCrawler is:

Recently, Yahoo! launched another crawler (named “Yahoo!-AdCrawler”) that also will crawl YSM advertiser landing pages. This crawler is designed to help us make relevance assessments of these pages, that helps us match ads to specific search queries.

Its perfectly safe for you to block it--and is highly recommended.

share|improve this answer
Thanks. The description on their site suggests it should not hammer a URL. I wonder if I have something set up wrongly on my site - which might cause other problems of course - in particular if no one else has seen this behaviour from AdCrawler – Kristen Sep 22 '10 at 7:42
Based on the comments I've seen elsewhere, it sounds like AdCrawler is generally a pain and not well received. There were some blog posts claiming they wasted gigabytes of bandwidth a month to AdCrawler specifically. – Andrew M. Sep 22 '10 at 20:18
I've come to that conclusion too. Its banned ... – Kristen Oct 3 '10 at 12:40

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.