Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I just updated my robots.txt file on a new site; Google Webmaster Tools reports it read my robots.txt 10 minutes before my last update.

Is there any way I can encourage Google to re-read my robots.txt as soon as possible?

UPDATE: Under Site Configuration | Crawler Access | Test robots.txt:

Home Page Access shows:

Googlebot is blocked from

FYI: The robots.txt that Google last read looks like this:

User-agent: *
Allow: /<a page>
Allow: /<a folder>
Disallow: /

Have I shot myself in the foot, or will it eventually read: http:///robots.txt (as it did the last time it read it)?

Any ideas on what I need to do?

share|improve this question
FYI: The site is new, and this message appears in Settings|Crawl Rate: "Your site has been assigned special crawl rate settings. You will not be able to change the crawl rate." – condiosluzverde Aug 18 '10 at 18:22
FYI: I found a posting in google groups that said google will read robots.txt "at least once a day" - can anyone confirm that? [google groups posting is here:… ] – condiosluzverde Aug 18 '10 at 22:08
FYI: 1 day has passed, and google has not yet read my updated robots.txt. – condiosluzverde Aug 19 '10 at 15:41
Same issue here, this is not a "feature"... – mate64 Feb 12 '13 at 11:54
up vote 20 down vote accepted

In case anyone else runs into this problem there is a way to force google-bot to re-download the robots.txt file.

Go to Health -> Fetch as Google [1] and have it fetch /robots.txt

That will re-download the file and google will also re-parse the file.

[1] in the previous Google UI it was 'Diagnostics -> Fetch as GoogleBot'.

share|improve this answer
Changed the accepted answer to this one, since it's way better than mine. – condiosluzverde Jul 9 '12 at 21:20
Unfortunately this will not work if your robots.txt is set to Disallow: /. Instead the fetch reports "Denied by robots.txt" :/. – studgeek Aug 2 '12 at 19:14
Next time add this line. Allow: /robots.txt – jrosell Sep 16 '12 at 16:52
I can't find 'Diagnostics', maybe the UI has changed? – David Riccitelli Dec 17 '12 at 7:28
Ok, it is now Health > Fetch as Google. – David Riccitelli Dec 17 '12 at 7:36

I know this is very old, but... If you uploaded the wrong robots.txt (disallowing all pages), you can try the following:

  • first correct your robots.txt to allow the correct pages, then
  • upload a sitemap.xml with your pages

as google tries to read the xml sitemap, it will check it agains robots.txt, forcing google to re-read your robots.txt.

share|improve this answer
This didn't work for me. It says the sitemap was blocked by the robots.txt – James Aug 29 '13 at 1:04

OK. Here is what I did, and within a few hours, Google re-read my robots.txt files.

We have 2 sites for every 1 site we run. Let's call them the canonical site ( and the bare-domain site (

We have our sites setup so that always returns a 301 redirecting to the

Once I setup both sites in Google Webmaster tools, told it that the is the canonical site, it soon after read the robots.txt file on the canonical site.

I don't really know why, but that's what happened.

share|improve this answer
I know this is old, but accepting your own answer is 100% legitimate – Mark Henderson Oct 25 '11 at 22:19

Shorten google scan interval for some days.

Also, I've seen there buttom to verify your robots.txt, this might force it to google, but I am not sure.

share|improve this answer
Can you be more specific? I see: Site Configuration | Crawler Access | Test robots.txt, but that tests the text you paste in the box, not your live robots.txt file - also, this is where it tells me when it was last downloaded. Where is the "verify" button you speak of? – condiosluzverde Aug 18 '10 at 18:24

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.