Have 10 computers, each have 12x2TB HDDs (currently) in raidZ2 (10+2) configuration, so, in the each computer i have one approx. 20TB volume.

Now, need those 10 separate computers (separate raid groups) join into one big volume.

What is the recommended solution?

I'm thinking about the FCoE (10GB ethernet). So, buying into each computer FCoE (10GB ethernet card) and - what need more on the hardware side? (probably another computer, FCoE switch? like Cisco Nexus?)

The main question is: what need to install and configure on each computer? Currently they have freebsd/raidz2, but it is possible change it into Linux/Solaris if needed.

Any helpful resource what talking about how to build a big volumes from smaller raid-groups (on the software side) is very welcomed. So, what OS, what filesystem, what software - etc.

In short: want get one approx. 200TB storage (in one filesystem) from already existing computers/storage. Don't need fast writes, but need good performance on reading data. (as a big fileserver), what will works transparently, so when storing data don't want care about onto what computer the data goes. (e.g. not 10 mountpoints - but one big logical filesystem).


  • 1
    For 200TB of storage, you'd be better off buying storage from a SAN vendor than tying to stitch 10 makeshift shelves of disks together – MDMarra Jun 3 '12 at 20:24
  • Does this 200TB volume have to be a block device, or just a filesystem with all the files in the one place? – Mark Henderson Jun 3 '12 at 20:26
  • @MDMarra: Thanx for your advice, but here is currently existing hw. The question is about how to "stitching" them together. – jm666 Jun 3 '12 at 20:27
  • one filesystem - probably don't need a block device (you mean SAN?). Simple want one 200TB filesystem, where can store files without need thinking about: this file into the filesystem on computer1 this into computer2. Therefore not want 10 different mount-points, but one logical filesystem). – jm666 Jun 3 '12 at 20:31
  • Rather than FCoE, ceph or gluster can just use ethernet or if you use FDR Infiniband cards (56Gbit) you can use IPoIB (in Ceph) or RDMA (Gluster). Performance should be stellar! – Matt Jul 25 '13 at 4:13

Ceph is a distributed object store and file system designed to provide excellent performance, reliability and scalability.

Nice overview of Ceph can be found in this webinar: Getting Started with Ceph

  • +1 for ceph. It's awesome. Put Infiniband cards in each machine and use IPoIB to network them. Then use that network as the cluster network. – Matt Jul 25 '13 at 4:11
  • To be taken seriously in Infiniband world ceph needs to implement native verbs interface. But still even with IPoIB on >= 12xQDR Infiniband ceph should be faster than 10G Ethernet interconnect. – SaveTheRbtz Jul 26 '13 at 18:33

glusterfs has striped volumes that could do this

  • +1 to GlusterFS – dyasny Jun 3 '12 at 21:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.