My server running apache is getting slammed by googlebot. I have sitemap index which points to several sitemaps with 50000 entries each. Google is requesting the sitemaps so aggressively (several times a minute) that my server is getting out of memory errors.
I am caching the sitemaps, but my web server still has to process these massive files rapidly.
What can I do?