Yahoo AdCrawler is re-trying some URLs repeatedly. The URLs are being given a 302 response code, so I suppose Yahoo should come back and try again "later", but "later" in my book doesn't mean that 7 specific URLs should be hit 3,000 times a day each.
The URLs are of the style:
The Query part may well be something our Client has added, but I mention it in case it rings any bells
We've banned AdCrawler in Robots.txt for now.
User-agent: Yahoo!-AdCrawler Disallow: /
If you have any thoughts, similar experiences, or ideas I'd appreciate them. Thanks.