I've started tracking user-agent strings on a website at the start of each session. Looking at the data for this month so far I'm seeing on search engine bot that keeps coming up a lot..
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
From 9/1/2011 to 9/13/2011 I've logged 2090 hits from this user-agent. From other search engines I'm tracking much lower numbers of hits...
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) - 353
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - 175
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) - 110
www.baidu.com seems to be a Chinese version of Google. Is there a way to throttle their bot? I don't mind them indexing us... in fact its probably a good thing as we have a large asian population utilizing the site, but they seem to be doing it a lot more.