-1

I have a wordpress blog on my own server, which used permanant links containing Chinese characters in urls like http://techblog.zellux.czm.cn/2008/03/ics-lab4-%E7%BB%8F%E9%AA%8C/.

Several months ago I changed all the urls with english descriptions like http://techblog.zellux.czm.cn/2009/05/page-coloring/, and installed a plugin to generated sitemap.xml automatically and submitted it to Google Webmaster Tools.

Now several months later I checked Google Webmaster Tools again, in Diagnostics->Web Crawl it said that it could not found 41 urls, all of which contained unicode of Chinese characters like the one before, and in the Linked From column they were all unavailable. As to Sitemaps->Sitemap details, it said that only 15 out of 115 URLs were indexed.

So my problem is how to make Google stop trying to crawl deprecated URLs but crawl only those specified in sitemap.xml?

closed as off topic by squillman, Sam Cogan, Jeff Atwood Jun 5 '09 at 9:58

Questions on Server Fault are expected to relate to server, networking, or related infrastructure administration within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here. If this question can be reworded to fit the rules in the help center, please edit the question.

1

You can use a robots.txt file to exclude specified pages from being crawled. It would look like this:

User-Agent: Googlebot
Disallow: /file-1
Disallow: /file-2
Disallow: /file-3

Create this in any text editor and upload to your root directory (or edit the existing robots.txt file if one is there).

Webmaster Tools also has a tool to generate a robots.txt file (Tools > Generate robots.txt)

Not the answer you're looking for? Browse other questions tagged or ask your own question.