Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I recently asked for 301 redirection on ServerFault and I didn't get a proper solution to my problem, but now I have a new idea: use the robots.txt to disallow certain URLs from my site to be "crawled".

My problem was simple: after a migration from a propietary, customised CMS to WordPress we had a lot of URLs that Google didn't find on the new site, and that went to 404 page. This is bad for our Pagerank and the searches, because Google still thinks those pages are alive.

We have a list of the URLs that don't work, and I tried to redirect to the good ones. The problem is, there are 20.000 of them, and there's no chance to solve the problem with a regular expression. We had to do 301 redirects ONE BY ONE, and that was a hell of a task.

But I was wondering: Could we just list those all bad URLs on our robots.txt with the "Disallow:" prefix, so Google does not index them? Is this a bad idea?

share|improve this question
Please don't use URL shortening, it's completely unnecessary. –  Dennis Williamson Dec 10 '10 at 14:24
Sorry. Won't do that again :( –  javipas Dec 10 '10 at 15:04

1 Answer 1

up vote 1 down vote accepted

If Google thinks that your 404 page is valid then you need to be returning a 404 response code on that page. Fix that, and the rest will be fine.

share|improve this answer
I'm sorry, but I don't understand your tip... Could you explain this with a little more detail, maybe some example? –  javipas Dec 10 '10 at 12:18
Your 404 page is not telling the client that it's a 404 page. Fix this. –  Ignacio Vazquez-Abrams Dec 10 '10 at 12:20
Ok, but... after doing that... should I do something with my robots.txt and disallow all the URLs that give a 404, or do I leave everything as is? BTW, thanks for your quick answers :P –  javipas Dec 10 '10 at 12:57
if the pages are not there (404) properly then they wont be indexed anyway. There is no need to add them to robots.txt –  JamesRyan Dec 10 '10 at 13:21
When the bots get the 404, they will mark the page as not found. –  Ignacio Vazquez-Abrams Dec 10 '10 at 13:21

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.