Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Our sitemap.xml file is becoming large. I'd like to prevent randoms from accessing the Sitemap.xml file and only allow google, yahoo and bing to access it. Kinda like a whitelist approach. Where can I find the IP addresses of search engine sitemap crawlers?

share|improve this question
    
Why? There should be nothing in your sitemap file that isn't public, so what difference does it make who accesses it? –  John Gardeniers Jul 14 '11 at 8:43

1 Answer 1

You're better off doing the filtering by user agent. You can search for the ones you want to allow here.

share|improve this answer
    
filtering based on User-Agent is a bad-idea as anyone can forge an HTTP Request with a fake User-Agent of allowed values and fetch the blocked resource. –  AbhishekKr Jul 14 '11 at 8:10

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.