up vote 1 down vote favorite
share [g+] share [fb]

My site has dynamic pages which look like 

www.mysite.com/?page&news_id=1

This is changed into www.mysite.com/news/1 by mod rewrite rules in my site.

How do I stop Google from indexing some of these dynamic pages?

link|improve this question
Can you be a bit more specific about what you mean by 'some'. Is it a well defined set of pages or some specific articles? – fredley Sep 15 '10 at 10:49
You may get a better answer at: webmasters.stackexchange.com – Rowland Shaw Sep 15 '10 at 10:49
It is certain specific articles. – CoderX Sep 15 '10 at 10:50
No, you don't want to block. You want to redirect all the old pages to the new URLs. You do this in your webserver configuration. – Alex Holst Apr 23 '11 at 19:57
feedback

migrated from stackoverflow.com Sep 16 '10 at 16:18

This question came from our site for professional and enthusiast programmers.

6 Answers

up vote 5 down vote

See robot.txt http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

link|improve this answer
The pages are generated dynamically – andyjohnson Sep 15 '10 at 10:53
2  
@Andy robots.txt can be generated dynamically too. – Rowland Shaw Sep 15 '10 at 10:57
@Rowland I guess that would work if its possible to enumerate all the pages to be blocked at the point where robots.txt is requested. It seemed to me that this might not be possible and/or might be inefficient if there are lots of pages. I've always considered robots.txt to be more useful for static content. – andyjohnson Sep 15 '10 at 11:10
@Andy I'd generate it when an article is added, removed or otherwise has its status changed – Rowland Shaw Sep 15 '10 at 11:24
You don't need to block each page explicitly.. you can use wildcards. – Chris Nava Sep 16 '10 at 16:55
feedback
up vote 4 down vote

Do not BLOCK those pages. Instead redirect the "old" pages to the new, clean ones...

link|improve this answer
feedback
up vote 3 down vote

Get create ROBOT file.

link|improve this answer
The pages are generated dynamically – andyjohnson Sep 15 '10 at 10:54
2  
@Andy robots.txt can be generated dynamically, too. – Rowland Shaw Sep 15 '10 at 10:57
feedback
up vote 3 down vote

What you want is a robots.txt file. If you're using Google Webmaster Tools, they even have a tool and instructions to help you generate one.

It's also quite simple to write one on your own. (Definitely simpler than using mod_rewrite) Both Wikipedia and the original spec have examples.

Sitemaps are useful, but robots.txt is still supported by a wider variety of crawlers... which, given your goal, seems to be something you'll want.

link|improve this answer
feedback
up vote 1 down vote

the best bet is to give google a sitemap of the URLs that you do actually want them to index - more information can be found about this http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156184

link|improve this answer
feedback
up vote 0 down vote

In pages that you don't want Google to index, check the user agent string in the http request. If it contains 'Googlebot' then you know its Google spidering your site and you can return a suitable http response code such as 401 Unauthorized.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard