Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a wordpress blog on my own server, which used permanant links containing Chinese characters in urls like http://techblog.zellux.czm.cn/2008/03/ics-lab4-%E7%BB%8F%E9%AA%8C/.

Several months ago I changed all the urls with english descriptions like http://techblog.zellux.czm.cn/2009/05/page-coloring/, and installed a plugin to generated sitemap.xml automatically and submitted it to Google Webmaster Tools.

Now several months later I checked Google Webmaster Tools again, in Diagnostics->Web Crawl it said that it could not found 41 urls, all of which contained unicode of Chinese characters like the one before, and in the Linked From column they were all unavailable. As to Sitemaps->Sitemap details, it said that only 15 out of 115 URLs were indexed.

So my problem is how to make Google stop trying to crawl deprecated URLs but crawl only those specified in sitemap.xml?

share|improve this question

closed as off topic by squillman, Sam, Jeff Atwood Jun 5 '09 at 9:58

Questions on Server Fault are expected to relate to professional server, networking, or related infrastructure administration within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

add comment

1 Answer

up vote 1 down vote accepted

You can use a robots.txt file to exclude specified pages from being crawled. It would look like this:

User-Agent: Googlebot
Disallow: /file-1
Disallow: /file-2
Disallow: /file-3

Create this in any text editor and upload to your root directory (or edit the existing robots.txt file if one is there).

Webmaster Tools also has a tool to generate a robots.txt file (Tools > Generate robots.txt)

share|improve this answer
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.