We need storage specifically for HPC Lustre failover setup, where it is must that two servers should share same block level storage to have failover configuration.

With very limited knowledge on hardware, I have the below understanding:

  • NAS can be used for shared storage, but there will be bottleneck for speed due to intermediate network.
  • SAN can be used, but it is costly to implement the solution and not really needed for Storage of 50-100TB.
  • If at all we find multiple iscsi ports to the storage enclosure, the storage can be used only by splitting i.e., works as two storage devices and the same storage can't be used by both the
    systems. (And one thing to remind here, in the lustre setup, both the servers would be only attached, but only one will be used (not sure, how it is possible, again need to check on this).
  • Having two virtual machines may be how we can do it. But, then, it is not really helpful for the purpose of failover, as the physical machine would be only one.

But, while posting the question, I am thinking, may be we can compromise on speed in NAS, if we try having one directly attached server (primary) and the other attached via network (failover), so we face slowness only when the primary stops working.

  • Have you looked at GFS2? Mar 14, 2016 at 10:44
  • @MichaelHampton We have no concept of using different file system, as we are already using lustre file system (lustre or zfs). But connecting to the storage device and its discovery in /dev/ in both the servers is what I am hoping for.
    – GP92
    Mar 14, 2016 at 10:52
  • 1
    Have you actually attempted to set up a proof of concept, or at least looked through the documentation? Lustre really doesn't care very much how you connect to the underlying storage, so you can do whatever gets you the bandwidth you need. Mar 14, 2016 at 11:02
  • @MichaelHampton Oh really!! I had a setup of lustre without failover, only did disk speed testing with bonnie++. I have gone through the documentation, but bits and pieces. May I ask, how it works almost the same on network or if directly attached, will it do any caching or kind of asynchronous interaction? Thanks for letting know.
    – GP92
    Mar 14, 2016 at 11:30
  • As this seems to be mainly about failover/high availability, I'm trying to understand how shared storage really helps you here, as your storage layer is the most likely component for failure in most architectures. You'd be better off getting storage replication going for a single server.
    – Joel Coel
    Mar 14, 2016 at 14:18

2 Answers 2


Michael is saying that lustre does not care. All you need for lustre is for a block device that appears in /dev.

You still need to pay attention to the other layers, e.g network. You will loose performance if your network links are slower than your disks. This will likely be the case over ethernet. You will also loose a small amount using iscsi rather than direct attach even with fast links. Its hard to really guess what will be your issues since it depends on your hardware. I think that's why its difficult to get or give a clear answer (I have seen your lustre discuss post, but I think this is a question related more generally to hardware and interconnects).


  • Yes, it's clear now what's Michael conveyed. And also I don't think it would be a big problem though, as I guess we can have primary node directly attached and the failover node on network, that could allow to have the maximum performance while primary node is up and slower performance while any disaster in the primary node.
    – GP92
    Mar 14, 2016 at 14:32

"If at all we find multiple iscsi ports to the storage enclosure, the storage can be used only by splitting i.e., works as two storage devices and the same storage can't be used by both the systems." isn't true, you can share an iSCSI (or any other block device) LUN with multiple machines but you NEED to use a cluster-aware filesystem with some form of distributed locking mechanism. Luckily these are easy to find but rarely the default with any operating system

If all you want is a bunch of shared disks shared between EXACTLY two hosts then I'd suggest you look at something like the HPE MSA2040 SAS disk system. As the name suggests it's SAS but it's setup to allow two servers to connect to it and both servers see the various logical disks you setup on the box. This way in the event of one server falling over the other can still see the same disks. You can set this up with OCFS2, GFS and a few other file systems - even NTFS with Server 2012/R2 when used with cluster services. It's pretty much the cheapest way you can do this unless you want to use FC/FCoE/iSCSI etc.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.