Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I would like to create a publicly accessible Google Apps site (i.e. users do not need to be authenticated to access the content) while maintaining a policy crawlers and bots exclusion with Robots.txt. Does anyone know how to do that?

share|improve this question
I don't specifically know much about Google Apps - is it stopping you from simply serving out a robots.txt "as usual"? – Jakob Borg Jan 18 '10 at 21:32
what are you trying to achieve as you cant make your apps site publicly accessible. – seanl Jan 19 '10 at 11:55
seanl, it's very possible to make a google site publicly accessible, see – Joannes Vermorel Jan 20 '10 at 17:08

robots.txt doesn't prevent interactive browsers from using the site. It is only used by robots like crawlers, feedreaders, recursive download tools (though the latter will let the user override it).

share|improve this answer
Absolutely, but it does not matter in my case. I just want the results not to show up in Google. – Joannes Vermorel Jan 20 '10 at 11:49
Yeah, but now it's obvious you can just write a two-line robots.txt that excludes everyone everywhere (/ and *). – Tobu Jan 20 '10 at 22:17

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.