3

We currently have 4 FC arrays containing sixteen 300GB drives each in a Sun StorageTek 6140 chassis connected to an old Sun X4170 M2 server running OpenIndiana. This has been working well for the past 10+ years. We are looking to upgrade. We don't have much to spend but considering the current environment we should not need to spend much to do better. The SAN serves AIX/HP-UX iSCSI clients and NFS for Linux VMs. I have come up with the following new configuration:

Dell R640/MD1420 cabling diagram

The Sun server will be replaced by a Dell R640 10-bay server running OmniOS with a SAS/NVMe backplane. The ZFS root pool will consist of two mirrored Dell 400-AJRR 300GB drives with two spares. The ZIL will be two Intel Optane DC P4800X 375GB U.2 drives. I haven't figured out what drives to use for ZFS cache devices yet. This server has 3 low-profile PCIe slots. Two will contain the Dell HBA355e adapter for connection to two Dell MD1420 storage arrays, each with 24 Dell 400-ATIN 600GB drives. Connecting the Dell HBA to the MD1420 will be done with Mini-SAS SFF-8644 cables. The MD1420 is old but we don't need anything fancy.

Based on the cabling configuration above, and assuming MPIO on the server, should I be able to achieve the following (I believe the answer is yes but want to confirm):

  1. If one EMM controller in any MD1420 fails, array will continue to function
  2. If one SFF cable fails, array will continue to function
  3. If one of the HBA355e controllers fails, array will continue to function
  4. Power supply, EMM controller, and SFF cables are all hot-swap on the arrays

Rather than daisy-chaining the two arrays, is there any benefit to connecting the second array directly to the HBA355e like the first array?

And, how are the array and drives managed? OME? Does that allow firmware updates to the MD1420 and Dell drives in the R640 and MD1420?

Improve this question
6
  • Would you consider a different OS for ZFS purposes? There's some inherent risk using a non-mainline solution for this today.
    – ewwhite
    Commented Jul 17 at 22:00
  • None of the hardware is cutting-edge. I am sure the SFP+ NIC and HBA are supported by OmniOS. What is the inherent risk? I'd prefer to keep the OS stack. As we are only serving up iSCSI and NFS, we are not demanding too much of the OS.
    – Albert Chin-A-Young
    Commented Jul 18 at 1:29
  • reddit.com/r/zfs/comments/1e4mq8c/…
    – ewwhite
    Commented Jul 18 at 8:09
  • As someone who fixes a lot of broken commercial and Solaris-based ZFS solutions as scale, your future self will appreciate using Linux or engaging a ZFS consultant for this.
    – ewwhite
    Commented Jul 18 at 8:11
  • ZFS on OmniOS is baked in at a much better level than ZFS on Ubuntu. Just looking at the installation instructions to get ZFS root onto Ubuntu doesn't look like fun. I know that all active ZFS development is occurring on OpenZFS and only then filtering down to OmniOS. Who knows how long this will continue. However considering that we have been running OpenIndiana in production for 10+ years and don't perform much software updates on the SAN, I'm not too worried.
    – Albert Chin-A-Young
    Commented Jul 18 at 17:22

1 Answer 1

Reset to default
1

The ideal cabling for a single host and two JBODs is two SAS chains:

Oracle's ZFS appliance cabling documentation is helpful.

Be sure to use some form of multipath to create your zpool devices. If you're interested in Linux, you can refer to elements of my ZFS High Availability guide at: https://github.com/ewwhite/zfs-ha/wiki

Oracle cabling diagram

Improve this answer
4
  • Thanks. This is very useful. I can cut my cabling in half.
    – Albert Chin-A-Young
    Commented Jul 18 at 17:09
  • Did this answer your question?
    – ewwhite
    Commented Jul 19 at 8:56
  • Yes, thank you.
    – Albert Chin-A-Young
    Commented Jul 20 at 5:41
  • 1
    Awesome guide you have on GitHub!
    – Nikita Kipriyanov
    Commented Jul 29 at 8:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .