How can I protected my sitemap index file and sitemap.xml files from leechers?

Question

I have a "content" website that some leechers and 419 scammers love to crawl agressively which also generates costs and performance issue. :( I have no choice: I need to prevent them to access the sitemap files and index. :(

I am doing the same as Facebook: I generate a sitemap index on the fly (/sitemap.php). I whitelisted the "good" crawlers with DNS reverse lookup (PHP) and agent check (Same as Stackoverflow). To prevent whitelisted engines to make the sitemap index content public I added that header (Stackoverflow forgot it):

header('Content-type: application/xml; charset="UTF-8"', true);
header('Pragma: no-cache');
header('X-Robots-Tag: NOARCHIVE');

Question 1: Am I missing something to protect the sitemap index file?

Question 2: The problem comes from the static sitemap (.xml.gz) files generated. How can I protect them? Even if they have a "hard to guess" name, they can be found easily with a simple google query (example: "site:stackoverflow.com filetype:xml") and I have a very limited access to .htaccess.

EDIT: This is not a server config issue. Prefered language is PHP.

EDIT 2: Sorry, this is pure programmatic question, but it has been transfered from SO and I cannot close/delete it. :(

Answer 1

You could always use a URL for the sitemap which will not be disclosed to anyone else apart from the engines that you'll explicitly submit to.

Have a look at http://en.wikipedia.org/wiki/Sitemaps

Answer 2

You could use robots.txt to disallow the file but you could also block the IP's. A simple way to do this is to look at the HTTP referrers in your web logs and write a cron job to take those IP's (by referrer) and add them to hosts.deny for your website.

Answer 3

How about not creating sitemap.php on the fly? Instead regenerate it once a day (or whatever makes sense) and serve it up as a static file. That way, even if 10,000 crawlers a day request it—so what?