Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Our sitemap.xml file is becoming large. I'd like to prevent randoms from accessing the Sitemap.xml file and only allow google, yahoo and bing to access it. Kinda like a whitelist approach. Where can I find the IP addresses of search engine sitemap crawlers?

share|improve this question
Why? There should be nothing in your sitemap file that isn't public, so what difference does it make who accesses it? – John Gardeniers Jul 14 '11 at 8:43

You're better off doing the filtering by user agent. You can search for the ones you want to allow here.

share|improve this answer
filtering based on User-Agent is a bad-idea as anyone can forge an HTTP Request with a fake User-Agent of allowed values and fetch the blocked resource. – AbhishekKr Jul 14 '11 at 8:10

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.