I have robots.txt that looks like this:

User-agent: *
Disallow: /system/
Disallow: /admin/
Disallow: /index.php

The obvious goal has been to prevent all the ugly URLs from being indexed, as they all begin with "/index.php".
But for some reason all URLs like /my-beautiful-sef-url-123 are listed under Crawl errors in Google Webmaster Tools with "URL restricted by robots.txt". (When I test such URL it yields Allowed for both Googlebot and Googlebot-Mobile.)

Can anyone help please?

link|improve this question
feedback

1 Answer

up vote 2 down vote accepted

Are the URLs like /my-beautiful-sef-url-123 real pages or do they do a redirect to another page? If they do a redirect, maybe try testing the redirected to page in Google Webmaster??

link|improve this answer
feedback

Your Answer

 
or
required, but never shown

Not the answer you're looking for? Browse other questions tagged or ask your own question.