Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I just updated my robots.txt file on a new site; Google Webmaster Tools reports it read my robots.txt 10 minutes before my last update.

Is there any way I can encourage Google to re-read my robots.txt as soon as possible?

UPDATE: Under Site Configuration | Crawler Access | Test robots.txt:

Home Page Access shows:

Googlebot is blocked from http://my.example.com/

FYI: The robots.txt that Google last read looks like this:

User-agent: *
Allow: /<a page>
Allow: /<a folder>
Disallow: /

Have I shot myself in the foot, or will it eventually read: http:///robots.txt (as it did the last time it read it)?

Any ideas on what I need to do?

share|improve this question
    
FYI: The site is new, and this message appears in Settings|Crawl Rate: "Your site has been assigned special crawl rate settings. You will not be able to change the crawl rate." –  jmsmcfrlnd Aug 18 '10 at 18:22
    
FYI: I found a posting in google groups that said google will read robots.txt "at least once a day" - can anyone confirm that? [google groups posting is here: groups.google.com/group/google_webmaster_help-indexing/… ] –  jmsmcfrlnd Aug 18 '10 at 22:08
    
FYI: 1 day has passed, and google has not yet read my updated robots.txt. –  jmsmcfrlnd Aug 19 '10 at 15:41
    
Same issue here, this is not a "feature"... –  cept0 Feb 12 '13 at 11:54

4 Answers 4

up vote 16 down vote accepted

In case anyone else runs into this problem there is a way to force google-bot to re-download the robots.txt file.

Go to Health -> Fetch as Google [1] and have it fetch /robots.txt

That will re-download the file and google will also re-parse the file.

[1] in the previous Google UI it was 'Diagnostics -> Fetch as GoogleBot'.

share|improve this answer
    
Changed the accepted answer to this one, since it's way better than mine. –  jmsmcfrlnd Jul 9 '12 at 21:20
9  
Unfortunately this will not work if your robots.txt is set to Disallow: /. Instead the fetch reports "Denied by robots.txt" :/. –  studgeek Aug 2 '12 at 19:14
3  
Next time add this line. Allow: /robots.txt –  jrosell Sep 16 '12 at 16:52
    
I can't find 'Diagnostics', maybe the UI has changed? –  David Riccitelli Dec 17 '12 at 7:28
2  
Ok, it is now Health > Fetch as Google. –  David Riccitelli Dec 17 '12 at 7:36

I know this is very old, but... If you uploaded the wrong robots.txt (disallowing all pages), you can try the following:

- first correct your robots.txt to allow the correct pages, then
- upload a sitemap.xml with your pages

as google tries to read the xml sitemap, it will check it agains robots.txt, forcing google to re-read your robots.txt.

share|improve this answer
    
This didn't work for me. It says the sitemap was blocked by the robots.txt –  James Aug 29 '13 at 1:04

OK. Here is what I did, and within a few hours, Google re-read my robots.txt files.

We have 2 sites for every 1 site we run. Let's call them the canonical site (www.mysite.com) and the bare-domain site (mysite.com).

We have our sites setup so that mysite.com always returns a 301 redirecting to the www.mysite.com.

Once I setup both sites in Google Webmaster tools, told it that the www.mysite.com is the canonical site, it soon after read the robots.txt file on the canonical site.

I don't really know why, but that's what happened.

share|improve this answer
    
Please pardon me for accepting my own answer, however there aren't any other answers that solved my issue, and I like to have 100% acceptance rate - thus, I am going to accept this answer. –  jmsmcfrlnd Sep 9 '10 at 2:25
3  
I know this is old, but accepting your own answer is 100% legitimate –  Mark Henderson Oct 25 '11 at 22:19

Shorten google scan interval for some days.

Also, I've seen there buttom to verify your robots.txt, this might force it to google, but I am not sure.

share|improve this answer
    
Can you be more specific? I see: Site Configuration | Crawler Access | Test robots.txt, but that tests the text you paste in the box, not your live robots.txt file - also, this is where it tells me when it was last downloaded. Where is the "verify" button you speak of? –  jmsmcfrlnd Aug 18 '10 at 18:24

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.