1

Ok, so I have a bit of an odd problem. Our site is crawled by google a LOT. There are a few reasons for this, that aren't really easy to change. One of which being you can access localized versions of our site via ita.example.com (italian) and fra.example.com (french) We do 301 redirects if not logged in, and the content wont be localized (think forum posts) but this still causes a large amount of crawling due to google thinking our 50 subdomains are all different sites (not the greatest idea, but before my time and not easily changeable)

I dont want to flat out rate limit google, which I know I can do in the webmaster tools or the robots.txt file

However, 7 year old forum posts don't really change, and dont need to be crawled daily, for 6 of the subdomains.

Now I know I could set an "expires" on the page, but then that would keep it from showing up in google search results, and I could block it with a meta tag, but that would do the same thing.

Is there any way I can specify to google "This is an old page, dont crawl it more than once a year" or something?

If there isnt a way to do that, is there any way to put a rate limit on certain directories?

3
  • This is more of a pointer than an answer, but is there some way you could get around this with a sitemap file?
    – Flup
    May 3, 2013 at 15:02
  • 1
    This looks like a good question for our sister site Webmasters. May 3, 2013 at 15:57
  • 2
    Since when does Google not index pages with Expires headers? I use them on every resource on my websites and I still get indexed.
    – Chris S
    May 3, 2013 at 16:02

2 Answers 2

1

sitemap.xml files (http://en.wikipedia.org/wiki/Sitemaps) include a <changefreq> element that indicates how frequently a search engine should check it for updates. Creating a sitemap and setting it to monthly, yearly, or never for those URLs should reduce the frequency with which Google crawls those URLs.

5
  • Thanks for the fast response. I was hoping to avoid that, as it would have been easier to include a meta tag on the page itself via php (if post > 1 year old changefreq=never) kind of logic... We have a HUGE site, so its going to be an undertaking
    – Excolo
    May 3, 2013 at 15:07
  • Sitemaps have other benefits that often make them well worth implementing. If your site is dynamically driven it may be a lot less work than you'd think...
    – ceejayoz
    May 3, 2013 at 15:08
  • It is all dynamically driven, all pages are php and go back through the same index file. I know that we had something a long time ago to generate sitemaps (that of course all current devs didnt work on, so have no real clue) and it generated a ton (size limit uncompressed is 50mb, so we have an "index" pointing to others, its really odd as im no seo expert)
    – Excolo
    May 3, 2013 at 15:11
  • also, another problem is, we would need 50 of them, one for each subdomain, as google requires absolute paths (or at least last time i checked)
    – Excolo
    May 3, 2013 at 15:13
  • Yeah, sitemap files have a size limit, so you often see an index file that points to a bunch of other chunked lists of URLs.
    – ceejayoz
    May 3, 2013 at 15:13
1

You can install the nofollow meta tag

< META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW" >

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .