up vote 1 down vote favorite
share [g+] share [fb]

I blocked a page in robots.txt under User-agent: *, and tried to do a manual removal of that URL from Google's cache in the webmasters tools. Google said it wasn't being blocked in my robots.txt, so I then blocked it specifically under User-agent: GoogleBot and tried removing it again and this time it worked. Does that mean Google doesn't respect User-agent: * or what?

link|improve this question
feedback

3 Answers

up vote 1 down vote

GoogleBot should properly follow rules for User-agent: *, however it will not update it's cache until it GoogleBot has crawled your site again, which takes some time. Google doesn't release the exact minimum time between crawls, but I'd speculate it's about 15 to 30 minutes.

link|improve this answer
The problem is it doesn't update its cache every time it crawls and sees my robots. That's why I'm trying to expedite the process by manual removal. GoogleBot seems to crawl at diff speeds depending on a lot of factors but according to my access logs it's only about once or twice a day for my site – user40696 Apr 25 '10 at 3:23
The interfaces available to the public only requests that GoogleBot crawls sites in a certain way, you can't control exactly what it does, and there is no way at all of directly interacting with the cache/index itself. Also, I'm fairly certain the more you request, the longer it is between re-crawls. – Chris S♦ Apr 25 '10 at 3:45
1  
You can interact with the cache/index, in the webmasters tools, click Site configuration>Crawler access>Remove URL and enter the URL. You can remove a page from just cache or both cache and search results.. – user40696 Apr 25 '10 at 3:55
I just tried this whole thing again on another page and the same thing happened. Google denied my request when under User-agent: *: "Your request has been denied because the webmaster of the site hasn't applied the appropriate robots.txt file or meta tags to block us from indexing or archiving this page. Please work with the webmaster of this site or select an alternate removal option from the webpage removal request tool." And when i put it under GoogleBot in robots.txt it suddenly accepted it for removal. So my question still stands. – user40696 Apr 25 '10 at 20:23
feedback
up vote 0 down vote

Incoming links will also make the url indexed by google. It wouldn't have an index of the actual content

link|improve this answer
feedback
up vote 0 down vote

If webmaster tools said it wasn't being blocked by robots.txt, I'd say you must have had an error in your robots.txt file. Google definitely does honor User-Agent so something else must be wrong. You can check out robots.txt in some detail via the help pages under Google Webmaster Tools.

Index removal takes 24-48 hours in my experience, although I have seen it happen in as little as 5 hours.

link|improve this answer
I didn't make any changes to my robots.txt after it didn't work other than putting the line that blocked the page under GoogleBot instead of *. Plus I did check that robots.txt was valid in the tools at the time. So that's not it. So it was either a fluke or GoogleBot doesn't always honor *. As for how long it takes, that's irrelevant because it put my request in the 'denied' section - meaning it processed it - which is what spurred me to come ask about this issue here. – user40696 Apr 25 '10 at 18:27
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.