I just updated my robots.txt file on a new site; Google Webmaster Tools reports it read my robots.txt 10 minutes before my last update.

Is there any way I can encourage Google to re-read my robots.txt as soon as possible?

UPDATE: Under Site Configuration | Crawler Access | Test robots.txt:

Home Page Access shows:

Googlebot is blocked from http://my.example.com/

FYI: The robots.txt that Google last read looks like this:

User-agent: *
Allow: /<a page>
Allow: /<a folder>
Disallow: /

Have I shot myself in the foot, or will it eventually read: http:///robots.txt (as it did the last time it read it)?

Any ideas on what I need to do?

  • FYI: The site is new, and this message appears in Settings|Crawl Rate: "Your site has been assigned special crawl rate settings. You will not be able to change the crawl rate." – qxotk Aug 18 '10 at 18:22
  • FYI: I found a posting in google groups that said google will read robots.txt "at least once a day" - can anyone confirm that? [google groups posting is here: groups.google.com/group/google_webmaster_help-indexing/… ] – qxotk Aug 18 '10 at 22:08
  • FYI: 1 day has passed, and google has not yet read my updated robots.txt. – qxotk Aug 19 '10 at 15:41
  • Same issue here, this is not a "feature"... – mate64 Feb 12 '13 at 11:54

In case anyone else runs into this problem there is a way to force google-bot to re-download the robots.txt file.

Go to Health -> Fetch as Google [1] and have it fetch /robots.txt

That will re-download the file and google will also re-parse the file.

[1] in the previous Google UI it was 'Diagnostics -> Fetch as GoogleBot'.

  • 11
    Unfortunately this will not work if your robots.txt is set to Disallow: /. Instead the fetch reports "Denied by robots.txt" :/. – studgeek Aug 2 '12 at 19:14
  • 3
    Next time add this line. Allow: /robots.txt – jrosell Sep 16 '12 at 16:52
  • I can't find 'Diagnostics', maybe the UI has changed? – David Riccitelli Dec 17 '12 at 7:28
  • 2
    Ok, it is now Health > Fetch as Google. – David Riccitelli Dec 17 '12 at 7:36
  • Not working for me when I try to fetch robots.txt. ERROR: "The page could not be crawled at this time because it is blocked by the most recent robots.txt file Googlebot downloaded. Note that if you recently updated the robots.txt file, it may take up to two days before it's refreshed. You can find more information in the Help Center article about robots.txt." – Indrek Feb 18 '13 at 15:16

I know this is very old, but... If you uploaded the wrong robots.txt (disallowing all pages), you can try the following:

  • first correct your robots.txt to allow the correct pages, then
  • upload a sitemap.xml with your pages

as google tries to read the xml sitemap, it will check it agains robots.txt, forcing google to re-read your robots.txt.

  • This didn't work for me. It says the sitemap was blocked by the robots.txt – James Aug 29 '13 at 1:04

OK. Here is what I did, and within a few hours, Google re-read my robots.txt files.

We have 2 sites for every 1 site we run. Let's call them the canonical site (www.mysite.com) and the bare-domain site (mysite.com).

We have our sites setup so that mysite.com always returns a 301 redirecting to the www.mysite.com.

Once I setup both sites in Google Webmaster tools, told it that the www.mysite.com is the canonical site, it soon after read the robots.txt file on the canonical site.

I don't really know why, but that's what happened.

  • 3
    I know this is old, but accepting your own answer is 100% legitimate – Mark Henderson Oct 25 '11 at 22:19

After have the same problem I sucessfuly made google reread my robots.txt file by submiting on this url:



Shorten google scan interval for some days.

Also, I've seen there buttom to verify your robots.txt, this might force it to google, but I am not sure.

  • Can you be more specific? I see: Site Configuration | Crawler Access | Test robots.txt, but that tests the text you paste in the box, not your live robots.txt file - also, this is where it tells me when it was last downloaded. Where is the "verify" button you speak of? – qxotk Aug 18 '10 at 18:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.