We are preparing to replace our storage servers (iSCSI+NFS). The current servers are Debian Wheezy using mdadm+lvm2 for storage, and failover using drbd and heartbeat (never got heartbeat to work).
For our replacement servers, I would like to use ZFS, but it has the limitation of not being able to reconfigure the raid set live. The server would be only partially populated initially. To add a drive, would need to export the whole file system, reconfigure, then import the file system.
I was originally planning on a pure FreeBSD system, using HAST+CARP to handle the nodes. HAST can only run on GEOM devices, which leaves out a zpool, so it would probably have to be run on a per-drive basis. There is a limit to the number of HAST devices on a system, but I have not been able to find out what this limit is.
Instead, I have come up with what could be a total kludge or could be a good answer. Here is the proposed system. It has one enterprise grade SSD for the OS, and 25 available hotswap bays for data.
I build the machine using Linux+mdadm. 2x120G SSD, RAID-1 and 8x500G SSD as RAID-6. Install xen, and build a FreeBSD virtual that has the the RAID-1 for operating system and the RAID-6 as a device for a zpool. This virtual is the storage server.
This gives the benefits of mdadm, zfs, hast and carp, at the cost of an additional layer (xen) using resources. HAST would run on the RAID-6 to the secondary machine (built exactly the same).
I'd love any feedback, from "you are an idiot" to "sounds ok to me," but preferably with reasoning behind it.