Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I host a few git repositories at git.nomeata.de using gitweb (and gitolite). Occasionally, a search engine spider comes along and begins to hammer the interface. While I generally do want my git repositories to show up in search engines, I do not want to block them completely. But they should not invoke expensive operations such as snapshotting the archive, searching or generating diffs.

What is the “best” robots.txt file for such an installation?

share|improve this question

closed as primarily opinion-based by HopelessN00b Mar 14 at 16:29

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.

1 Answer 1

I guess this makes a good community wiki. Please extend this robots.txt if you think it can be improved:

User-agent: * 
Disallow: /*a=search*
Disallow: /*/search/*
Disallow: /*a=blobdiff*
Disallow: /*/blobdiff/*
Disallow: /*a=commitdiff*
Disallow: /*/commitdiff/*
Disallow: /*a=snapshot*
Disallow: /*/snapshot/*
Disallow: /*a=blame*
Disallow: /*/blame/*
share|improve this answer
    
I've found that depending on how gitweb is configured, these URLs are not enough; e.g. in my installation, I needed to add for each /.*a=xxx* an entry of the type /*/xxx/*. –  iustin Mar 14 at 12:43

Not the answer you're looking for? Browse other questions tagged or ask your own question.