I blocked a page in robots.txt under User-agent: *, and tried to do a manual removal of that URL from Google's cache in the webmasters tools. Google said it wasn't being blocked in my robots.txt, so I then blocked it specifically under User-agent: GoogleBot and tried removing it again and this time it worked. Does that mean Google doesn't respect User-agent: * or what?
3 Answers
GoogleBot should properly follow rules for User-agent: *, however it will not update it's cache until it GoogleBot has crawled your site again, which takes some time. Google doesn't release the exact minimum time between crawls, but I'd speculate it's about 15 to 30 minutes.
-
The problem is it doesn't update its cache every time it crawls and sees my robots. That's why I'm trying to expedite the process by manual removal. GoogleBot seems to crawl at diff speeds depending on a lot of factors but according to my access logs it's only about once or twice a day for my site Apr 25, 2010 at 3:23
-
The interfaces available to the public only requests that GoogleBot crawls sites in a certain way, you can't control exactly what it does, and there is no way at all of directly interacting with the cache/index itself. Also, I'm fairly certain the more you request, the longer it is between re-crawls.– Chris SApr 25, 2010 at 3:45
-
1You can interact with the cache/index, in the webmasters tools, click Site configuration>Crawler access>Remove URL and enter the URL. You can remove a page from just cache or both cache and search results.. Apr 25, 2010 at 3:55
-
I just tried this whole thing again on another page and the same thing happened. Google denied my request when under User-agent: *: "Your request has been denied because the webmaster of the site hasn't applied the appropriate robots.txt file or meta tags to block us from indexing or archiving this page. Please work with the webmaster of this site or select an alternate removal option from the webpage removal request tool." And when i put it under GoogleBot in robots.txt it suddenly accepted it for removal. So my question still stands. Apr 25, 2010 at 20:23
Incoming links will also make the url indexed by google. It wouldn't have an index of the actual content
If webmaster tools said it wasn't being blocked by robots.txt, I'd say you must have had an error in your robots.txt file. Google definitely does honor User-Agent so something else must be wrong. You can check out robots.txt in some detail via the help pages under Google Webmaster Tools.
Index removal takes 24-48 hours in my experience, although I have seen it happen in as little as 5 hours.
-
I didn't make any changes to my robots.txt after it didn't work other than putting the line that blocked the page under GoogleBot instead of *. Plus I did check that robots.txt was valid in the tools at the time. So that's not it. So it was either a fluke or GoogleBot doesn't always honor *. As for how long it takes, that's irrelevant because it put my request in the 'denied' section - meaning it processed it - which is what spurred me to come ask about this issue here. Apr 25, 2010 at 18:27