23

I just updated my robots.txt file on a new site; Google Webmaster Tools reports it read my robots.txt 10 minutes before my last update.

Is there any way I can encourage Google to re-read my robots.txt as soon as possible?

UPDATE: Under Site Configuration | Crawler Access | Test robots.txt:

Home Page Access shows:

Googlebot is blocked from http://my.example.com/

FYI: The robots.txt that Google last read looks like this:

User-agent: *
Allow: /<a page>
Allow: /<a folder>
Disallow: /

Have I shot myself in the foot, or will it eventually read: http:///robots.txt (as it did the last time it read it)?

Any ideas on what I need to do?

Improve this question
4
  • FYI: The site is new, and this message appears in Settings|Crawl Rate: "Your site has been assigned special crawl rate settings. You will not be able to change the crawl rate."
    – qxotk
    Aug 18, 2010 at 18:22
  • FYI: I found a posting in google groups that said google will read robots.txt "at least once a day" - can anyone confirm that? [google groups posting is here: groups.google.com/group/google_webmaster_help-indexing/… ]
    – qxotk
    Aug 18, 2010 at 22:08
  • FYI: 1 day has passed, and google has not yet read my updated robots.txt.
    – qxotk
    Aug 19, 2010 at 15:41
  • Same issue here, this is not a "feature"...
    – mate64
    Feb 12, 2013 at 11:54

5 Answers 5

Reset to default
25

In case anyone else runs into this problem there is a way to force google-bot to re-download the robots.txt file.

Go to Health -> Fetch as Google [1] and have it fetch /robots.txt

That will re-download the file and google will also re-parse the file.

[1] in the previous Google UI it was 'Diagnostics -> Fetch as GoogleBot'.

Improve this answer
7
  • 11
    Unfortunately this will not work if your robots.txt is set to Disallow: /. Instead the fetch reports "Denied by robots.txt" :/.
    – studgeek
    Aug 2, 2012 at 19:14
  • 3
    Next time add this line. Allow: /robots.txt
    – jrosell
    Sep 16, 2012 at 16:52
  • I can't find 'Diagnostics', maybe the UI has changed?
    – David Riccitelli
    Dec 17, 2012 at 7:28
  • 2
    Ok, it is now Health > Fetch as Google.
    – David Riccitelli
    Dec 17, 2012 at 7:36
  • Not working for me when I try to fetch robots.txt. ERROR: "The page could not be crawled at this time because it is blocked by the most recent robots.txt file Googlebot downloaded. Note that if you recently updated the robots.txt file, it may take up to two days before it's refreshed. You can find more information in the Help Center article about robots.txt."
    – Indrek
    Feb 18, 2013 at 15:16
4

I know this is very old, but... If you uploaded the wrong robots.txt (disallowing all pages), you can try the following:

  • first correct your robots.txt to allow the correct pages, then
  • upload a sitemap.xml with your pages

as google tries to read the xml sitemap, it will check it agains robots.txt, forcing google to re-read your robots.txt.

Improve this answer
1
  • This didn't work for me. It says the sitemap was blocked by the robots.txt
    – James
    Aug 29, 2013 at 1:04
2

After have the same problem I sucessfuly made google reread my robots.txt file by submiting on this url:

https://www.google.com/webmasters/tools/robots-testing-tool

Improve this answer
1

OK. Here is what I did, and within a few hours, Google re-read my robots.txt files.

We have 2 sites for every 1 site we run. Let's call them the canonical site (www.mysite.com) and the bare-domain site (mysite.com).

We have our sites setup so that mysite.com always returns a 301 redirecting to the www.mysite.com.

Once I setup both sites in Google Webmaster tools, told it that the www.mysite.com is the canonical site, it soon after read the robots.txt file on the canonical site.

I don't really know why, but that's what happened.

Improve this answer
1
  • 3
    I know this is old, but accepting your own answer is 100% legitimate
    – Mark Henderson
    Oct 25, 2011 at 22:19
0

Shorten google scan interval for some days.

Also, I've seen there buttom to verify your robots.txt, this might force it to google, but I am not sure.

Improve this answer
1
  • Can you be more specific? I see: Site Configuration | Crawler Access | Test robots.txt, but that tests the text you paste in the box, not your live robots.txt file - also, this is where it tells me when it was last downloaded. Where is the "verify" button you speak of?
    – qxotk
    Aug 18, 2010 at 18:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .