My server running apache is getting slammed by googlebot. I have sitemap index which points to several sitemaps with 50000 entries each. Google is requesting the sitemaps so aggressively (several times a minute) that my server is getting out of memory errors.

I am caching the sitemaps, but my web server still has to process these massive files rapidly.

What can I do?

  • In the Google Webmaster Tools, there is an option to reduce the crawling frequency. But please note that I have never used it, and I don't know if it only influences the crawling of the pages, or if it also influences how often the sitemaps are crawled. If you are going to test that, I would be grateful if you could report the results. – Binarus Sep 15 '18 at 20:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.