Ok, so I have a bit of an odd problem. Our site is crawled by google a LOT. There are a few reasons for this, that aren't really easy to change. One of which being you can access localized versions of our site via ita.example.com (italian) and fra.example.com (french) We do 301 redirects if not logged in, and the content wont be localized (think forum posts) but this still causes a large amount of crawling due to google thinking our 50 subdomains are all different sites (not the greatest idea, but before my time and not easily changeable)

I dont want to flat out rate limit google, which I know I can do in the webmaster tools or the robots.txt file

However, 7 year old forum posts don't really change, and dont need to be crawled daily, for 6 of the subdomains.

Now I know I could set an "expires" on the page, but then that would keep it from showing up in google search results, and I could block it with a meta tag, but that would do the same thing.

Is there any way I can specify to google "This is an old page, dont crawl it more than once a year" or something?

If there isnt a way to do that, is there any way to put a rate limit on certain directories?

  • This is more of a pointer than an answer, but is there some way you could get around this with a sitemap file? – Flup May 3 '13 at 15:02
  • 1
    This looks like a good question for our sister site Webmasters. – Michael Hampton May 3 '13 at 15:57
  • 2
    Since when does Google not index pages with Expires headers? I use them on every resource on my websites and I still get indexed. – Chris S May 3 '13 at 16:02

sitemap.xml files (http://en.wikipedia.org/wiki/Sitemaps) include a <changefreq> element that indicates how frequently a search engine should check it for updates. Creating a sitemap and setting it to monthly, yearly, or never for those URLs should reduce the frequency with which Google crawls those URLs.

| improve this answer | |
  • Thanks for the fast response. I was hoping to avoid that, as it would have been easier to include a meta tag on the page itself via php (if post > 1 year old changefreq=never) kind of logic... We have a HUGE site, so its going to be an undertaking – Excolo May 3 '13 at 15:07
  • Sitemaps have other benefits that often make them well worth implementing. If your site is dynamically driven it may be a lot less work than you'd think... – ceejayoz May 3 '13 at 15:08
  • It is all dynamically driven, all pages are php and go back through the same index file. I know that we had something a long time ago to generate sitemaps (that of course all current devs didnt work on, so have no real clue) and it generated a ton (size limit uncompressed is 50mb, so we have an "index" pointing to others, its really odd as im no seo expert) – Excolo May 3 '13 at 15:11
  • also, another problem is, we would need 50 of them, one for each subdomain, as google requires absolute paths (or at least last time i checked) – Excolo May 3 '13 at 15:13
  • Yeah, sitemap files have a size limit, so you often see an index file that points to a bunch of other chunked lists of URLs. – ceejayoz May 3 '13 at 15:13

You can install the nofollow meta tag


| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.