2

I host a few git repositories at git.nomeata.de using gitweb (and gitolite). Occasionally, a search engine spider comes along and begins to hammer the interface. While I generally do want my git repositories to show up in search engines, I do not want to block them completely. But they should not invoke expensive operations such as snapshotting the archive, searching or generating diffs.

What is the “best” robots.txt file for such an installation?

Improve this question

1 Answer 1

Reset to default
2

I guess this makes a good community wiki. Please extend this robots.txt if you think it can be improved:

User-agent: * 
Disallow: /*a=search*
Disallow: /*/search/*
Disallow: /*a=blobdiff*
Disallow: /*/blobdiff/*
Disallow: /*a=commitdiff*
Disallow: /*/commitdiff/*
Disallow: /*a=snapshot*
Disallow: /*/snapshot/*
Disallow: /*a=blame*
Disallow: /*/blame/*
Improve this answer
1
  • I've found that depending on how gitweb is configured, these URLs are not enough; e.g. in my installation, I needed to add for each /.*a=xxx* an entry of the type /*/xxx/*.
    – iustin
    Mar 14, 2015 at 12:43

Not the answer you're looking for? Browse other questions tagged .