Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I host a few git repositories at git.nomeata.de using gitweb (and gitolite). Occasionally, a search engine spider comes along and begins to hammer the interface. While I generally do want my git repositories to show up in search engines, I do not want to block them completely. But they should not invoke expensive operations such as snapshotting the archive, searching or generating diffs.

What is the “best” robots.txt file for such an installation?

share|improve this question

1 Answer 1

I guess this makes a good community wiki. Please extend this robots.txt if you think it can be improved:

User-agent: * 
Disallow: /*a=search*
Disallow: /*a=blobdiff*
Disallow: /*a=commitdiff*
Disallow: /*a=snapshot*
Disallow: /*a=blame*
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.