looking at my Apache access.log I see that crawlers tend to get old versions of pages and documents, like:
22.214.171.124 - - [10/Jun/2011:10:36:31 +0200] "GET /wiki/News?version=14 HTTP/1.1" 200 6073 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
I'd like them not to append the
?version=x suffix to URLs, so that they only get the most recent contents.
Is there a way to do this via the
robots.txt file (or other mechanisms that I don't know?