2

I am looking at buying a cost-effective machine for using openAttic, which is simply Ubuntu 14.04 64bit with ZFS.

The server I am looking at gives me 8 bays of 3.5", where I plan to put SAS 7.2k enterprise grade HDDs, probably 3Tb or 4Tb. I'm aiming for 18Tb to 24Tb of usable space, with 2 HDDs for redundancy. I'll get 32Gb of ECC RAM at least, but I will look at cost of 64Gb.

I am not experienced enough with ZFS, but initially my plan was to use a small boot and system partition, mirrored across all the drives, using ext4. Then using the remaining space for ZFS.

The server has an M.2 slot: Should I get an M.2 SSD there to use as the boot/system drive? OpenATTIC needs very little for the system partition, so it seems like a waste to get a premium SSD for that, when RAM and the SAS drives will be doing most of the work and will be the bottleneck.

I have read that ZFS likes to get direct access to the disks, so I wonder: Can ZFS can work well when you configure it to not use the whole disk? Or should I get the SSD and let ZFS take all the SAS disks?

If money is not a limitation, of course I get the SSD, but I am looking for "best value for money". Thanks!

2

For something of this scale, use a separate drive or pair of drives for the OS. Perhaps the M.2 SSD slot is the most appropriate.

Give ZFS the full disks and don't attempt to spread the OS across those drives.

For home-scale, don't listen to all of the suggestions to get a ZIL and L2ARC. The ZIL is only useful for synchronous writes, and the L2ARC is overrated. You have a good RAM count, and that's what you should maximize.

2

You have several choices:

  • Cheap USB 2.0 stick: not recommended because they are slow and the writes will kill most of them pretty quickly, exception: a read-only system like SmartOS (for ZFS) or ESXi (no ZFS, just as an example) which only writes at bootup or once an hour and keeps everything else in RAM or your data pools.
  • More expensive USB 3.0 stick with SLC memory or USB 3.0 HDD: Although not officially supported on most systems, it works pretty well. I would suggest mirroring these drives, which also enables seamless upgrades and backups easily.
  • SATA or M.2 disk/SSD: may waste valuable space, more expensive, but problem-free. I would suggest a mirrored pair if you can afford it. Best solution for most cases where space and money are not the most important constraints. Best choice would be with full power-loss protection (like Intel 730 and all S3xxx/DC3xxx versions).
  • NFS/iSCSI mount: interesting if you have several boxes with no space/ports left and you want to have central management/configuration. Boot via PXE, then map the network drives and use them. Almost all server NICs support this, but you need a separate server to provide the services.

My personal suggestion:

  • If you have just a normal filer setup and want to saturate 1Gbps: use 2 SLC USB sticks like the MX-ES models (external) or two small SATA Intel SSDs (internal)
  • If you have a larger setup, want high performance, quick change of failed disks, maybe have virtualisation or high uptime requests: use your normal enclosures with 2 Intel SSDs with power-loss protection
  • If you have/grow a large amount of machines and want to save time/money: consider either SmartOS (built for this use case) or use network boot via PXE

I have read that ZFS likes to get direct access to the disks, so I wonder: Can ZFS can work well when you configure it to not use the whole disk? Or should I get the SSD and let ZFS take all the SAS disks?

It is recommended to use whole disks for your pool:

For production systems, use whole disks rather than slices for storage pools for the following reasons:

  • Allows ZFS to enable the disk's write cache for those disks that have write caches. If you are using a RAID array with a non-volatile write cache, then this is less of an issue and slices as vdevs should still gain the benefit of the array's write cache.
  • For JBOD attached storage, having an enabled disk cache, allows some synchronous writes to be issued as multiple disk writes followed by a single cache flush allowing the disk controller to optimize I/O scheduling. Separately, for systems that lacks proper support for SATA NCQ or SCSI TCQ, having an enabled write cache allows the host to issue single I/O operation asynchronously from physical I/O.
  • The recovery process of replacing a failed disk is more complex when disks contain both ZFS and UFS file systems on slices.
  • ZFS pools (and underlying disks) that also contain UFS file systems on slices cannot be easily migrated to other systems by using zpool import and export features.
  • In general, maintaining slices increases administration time and cost. Lower your administration costs by simplifying your storage pool configuration model.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.