Currently, I'm tasked with progressively building a ZFS store. IOPS are not the priority, data security and capacity are.
In essence what we have is one brute of a machine, that contains an OS (FreeBSD most likely but OmniOS is a possible), 2 RAID-1 OS disks, and 4 SSDs, for caching and ZILs. RAM will start of at 16GB but I can go all the way up to 128GB if need be (and if accounting doesn't suffer a stroke). That in turn is connected to a JBOD chassis that can handle upto 45 drives. Each drive is 4TB.
I was considering incrementing by 9 drives, using a RAID-Z2 schema and then as needed, creating a new vDEV of another 9 drives until the 45 drive capacity is fulfilled, after which we would add another chassis that also handles another 45 drives. There is a limit of 3 or 4 connected chassis before a new similar configuration would need to be created.
Another possible option is to go up in 7s (RAID-Z2) and once we hit the 42 drive mark, we can slot in 3 drives to be used at any moment by any of the vdevs if a drive pushes up the daisies.
The Architecture is something as follows:
Storage Server | | | ------------------------------------------------ | | | JBOD 1 ... JBOD 2 JBOD 3 -vdev 1.1 (9 drives) -vdev 2.1 (9 drives) -vdev 3.1 (9 drives) -vdev 1.2 (9 drives) -vdev 2.2 (9 drives) -vdev 3.2 (9 drives) -vdev 1.3 (9 drives) -vdev 2.3 (9 drives) -vdev 3.3 (9 drives) -vdev 1.4 (9 drives) -vdev 2.4 (9 drives) -vdev 3.4 (9 drives) -vdev 1.5 (9 drives) -vdev 2.5 (9 drives) -vdev 3.5 (9 drives)
On a side note, I'd also highly appreciate any input on FreeBSD vs Illumian vs OmniOS with respect to ZFS. I'm fairly fluent on FreeBSD and I've played a bit with OmniOS but no working knowledge on Illumian.
Referencing Nex7's blog post point 9, we're building our VDEV's on 9 disks, 2 are for parity, so we're still in the clear, but we could I believe exchange that for a 15 vdev with 2 for parity since we're using RAIDZ2.
Point made on the SATA drives, switching to SAS will not be a problem. The SSD drives are Intel server drives, but I'll be checking. In all, because we have a few dozen boxes running 100% on the same hardware, I'm not worried about compatibility being an issue.
The reason we did not got with Linux is because we're not 100% certain of how decent and stable the implementation is. If it were stable...
What advantage does OpenIndiana have over FreeBSD for ZFS? What is the advantage of Napp-it?
How would you suggest we deal with the storage head HA issue?