up vote 1 down vote favorite
2
share [g+] share [fb]

I just updated my robots.txt file on a new site; Google Webmaster Tools reports it read my robots.txt 10 minutes before my last update.

Is there any way I can encourage Google to re-read my robots.txt as soon as possible?

UPDATE: Under Site Configuration | Crawler Access | Test robots.txt:

Home Page Access shows:

Googlebot is blocked from http://my.example.com/

FYI: The robots.txt that Google last read looks like this:

User-agent: *
Allow: /<a page>
Allow: /<a folder>
Disallow: /

Have I shot myself in the foot, or will it eventually read: http:///robots.txt (as it did the last time it read it)?

Any ideas on what I need to do?

link|improve this question
92% accept rate
FYI: The site is new, and this message appears in Settings|Crawl Rate: "Your site has been assigned special crawl rate settings. You will not be able to change the crawl rate." – jmsmcfrlnd Aug 18 '10 at 18:22
FYI: I found a posting in google groups that said google will read robots.txt "at least once a day" - can anyone confirm that? [google groups posting is here: groups.google.com/group/google_webmaster_help-indexing/… ] – jmsmcfrlnd Aug 18 '10 at 22:08
FYI: 1 day has passed, and google has not yet read my updated robots.txt. – jmsmcfrlnd Aug 19 '10 at 15:41
feedback

4 Answers

up vote 2 down vote

In case anyone else runs into this problem there is a way to force google-bot to re-download the robots.txt file.

Go to Diagnostics -> Fetch as GoogleBot and have it fetch /robots.txt

That will re-download the file and google will also re-parse the file.

link|improve this answer
feedback
up vote 1 down vote accepted

OK. Here is what I did, and within a few hours, Google re-read my robots.txt files.

We have 2 sites for every 1 site we run. Let's call them the canonical site (www.mysite.com) and the bare-domain site (mysite.com).

We have our sites setup so that mysite.com always returns a 301 redirecting to the www.mysite.com.

Once I setup both sites in Google Webmaster tools, told it that the www.mysite.com is the canonical site, it soon after read the robots.txt file on the canonical site.

I don't really know why, but that's what happened.

link|improve this answer
Please pardon me for accepting my own answer, however there aren't any other answers that solved my issue, and I like to have 100% acceptance rate - thus, I am going to accept this answer. – jmsmcfrlnd Sep 9 '10 at 2:25
I know this is old, but accepting your own answer is 100% legitimate – Mark Henderson♦ Oct 25 '11 at 22:19
feedback
up vote 0 down vote

Shorten google scan interval for some days.

Also, I've seen there buttom to verify your robots.txt, this might force it to google, but I am not sure.

link|improve this answer
Can you be more specific? I see: Site Configuration | Crawler Access | Test robots.txt, but that tests the text you paste in the box, not your live robots.txt file - also, this is where it tells me when it was last downloaded. Where is the "verify" button you speak of? – jmsmcfrlnd Aug 18 '10 at 18:24
feedback
up vote 0 down vote

I know this is very old, but... If you uploaded the wrong robots.txt (disallowing all pages), you can try the following:

- first correct your robots.txt to allow the correct pages, then
- upload a sitemap.xml with your pages

as google tries to read the xml sitemap, it will check it agains robots.txt, forcing google to re-read your robots.txt.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.