7

I need to build a solution to host internal git repositories. It needs to supports hundreds of thousands (or more) repositories.

I plan on using multiple "dumb" servers with a shared storage, so basically when a client is trying to access a repository - it will be redirected by the load-balancer to any of the available servers. Any change to the repository - will be replicated across all nodes.

My first thought was to use GlusterFS for that, but I've read it doesn't handle well with small files. I'm also thinking of replicating everything myself using DRBD, but this requires more setup and seems more complicated when comparing to GlusterFS.

Which one of the two provides better performances? Basically the problem I'm trying to solve is that when any of the servers goes down - I want others to still be able to serve the data.

Improve this question

5 Answers 5

Reset to default
5

This is a classic scale-out use case, and IMO GlusterFS should fit the bill. You can give it a try - just bring a few VMs up, set up a few bricks to be used for repository storage and run a stress test.

DRBD is not an option here anyway - it doesn't scale. If anything, I'd look at other object storage projects (Swift for example), if Gluster doesn't work well enough, but none of them are extremely performance oriented

Improve this answer
4
  • 1
    It seems like GitHub are using DRBD for their storage (I guess it does scale), but it seemed like an overkill IMO, and I do prefer a distributed file-system solution where everything "just works".
    – Gilad Novik
    Jan 28, 2014 at 16:49
  • Well, the fact that someone uses it, doesn't mean it scales. It only means they have some failover clusters using DRBD, which is not the same as multiple copy active/active file access nodes you can get with gluster/swift/ceph/etc
    – dyasny
    Jan 28, 2014 at 17:23
  • Does DFS's performance drop with the an increasing number of nodes?
    – CMCDragonkai
    Jun 20, 2014 at 5:34
  • what does DFS have to do with gluster/drbd?
    – dyasny
    Jun 22, 2014 at 4:58
4

I had a similar set up for cyrus mail servers in which gluster has proven not to be able to handle the load during stress tests. we originally chose gluster because it was simple but had to move back to drbd. however as dyasny highlighted, drbd activs/active cfg is not advisable (not to mention the pain). if you need all servers to mount the shared storage at the same time drbd is not an option. Another thing you may want to look at is clvmd with lock manager. lvm2 now have support for raid type lv and use man code for that purpose. this mixed with the appropriate filesystem (some cluster aware ones if needed) can be an option. however I never tested it myself in production, only as PoC.

Improve this answer
1
  • 1
    Just one thing I'd like to add, cluster didn't cope in my setup, but keep in mind it was a 2 nodes cluster. Gluster would have behaved better with more nodes.
    – alxgomz
    Jan 30, 2014 at 7:35
3

The problem you will have with Gluster is mainly latency between the nodes. I would advise you give it a try and see if it handles your workload. If your servers are are powerful enough and have a fast interconnect you should get a fairly good performance. Also you might want to try the built-in NFS server, which, talking from my experience, handles small files a little bit faster.

But if you need your solution as fast as possible GlusterFS probably isn't for you. Other options would be commercial products like Git Clustering(haven't tested that, though) or you could build your own solution with free tools like gitmirror.

Improve this answer
2
  • Actually performances is not my #1 priority (few more seconds are still ok), but stability and replication is. Which NFS server were you referring to?
    – Gilad Novik
    Jan 28, 2014 at 16:46
  • GlusterFS has a built-in NFS server. Performance is better but you have to handle the failover on the clients yourself. On the other hand you save yourself the need of installing the FUSE-client for access to the GlusterFS servers.
    – Izzy
    Jan 28, 2014 at 21:13
2

If you are considering Gluster (which in my experience is usable with many small files but YMMV), you might want to have a look at Ceph as well http://ceph.com/

Improve this answer
2
  • My problem with Ceph is that it has a meta-data server, which seems to actually hurt performances since every data access will also require accessing the meta-data server. Am I wrong?
    – Gilad Novik
    Jan 28, 2014 at 16:42
  • metadata servers can be duplicated too, and can run on the same nodes as object store servers. You'll need to fetch metadata at any point, whether you fetch it from the object store or from the metadata server, so it shouldn't significantly impact performance. I haven't actually tried any of this though.
    – Wouter Verhelst
    Nov 26, 2017 at 9:08
0

I would also recomend Glustet ad the easyest, and fastest to setup , configure. Also it scales out very very well. The problem is speed, and my 2c ia that you need to choose between easy configuration and easy scaleout, and between some technical dificulties (some fancy drbd / ocfs2 / Glances block storage) with the gain of speed. But..how much speed you gain? Yoi need to make some stress test and choose..

Improve this answer
3
  • Assuming I want to extend the storage, can I simply add another VM to the list and it will replicate to it, or will it completely block access to entire storage until this VM is ready to be used in the cluster? Is rebalancing happen asynchronously?
    – Gilad Novik
    Jan 28, 2014 at 16:48
  • Gluster is brickable, so any new brick will be added to the one, single, uniqe brick..
    – x86fantini
    Jan 28, 2014 at 16:55
  • So I guess if I want the fastest way to add a brick without stopping the cluster for too long, I should only use fixed layout and not migrate data?
    – Gilad Novik
    Jan 28, 2014 at 17:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .