If you are running a 3 or more node cluster with cman/corosync and you have some sort of shared block storage (SAN) connected only to some nodes in the cluster how can you enable CLVM on that storage device?

In my example I'm running a 3 node cluster, there 2 nodes are "main workload" and 1 node are used for backups & archives. Main nodes are connected with FC HBA with multipath to a SAN. Everything works fine, I was able to initialize PV on that device and can see it on both nodes:

  PV                 VG      Fmt  Attr PSize   PFree  
  /dev/mapper/mpathd san-vg1 lvm2 a--  500.00g 500.00g

Now, to create a clustered VG, I've enabled fencing on all 3 nodes in my cluster (with IPMI if that matters).

fence domain
member count  3
victim count  0
victim now    0
master nodeid 1
wait state    none
members       1 2 3 

I've also created a clustered VG (vgcreate -c)on that PV and that one works fine too:

  VG      #PV #LV #SN Attr   VSize   VFree  
  san-vg1   1   0   0 wz--nc 500.00g 500.00g

But I cannot create LV on that VG now, because 3rd node in the cluster doesn't have FC HBA, and don't "see" that PV at all:

  Error locking on node main-stor1: Volume group for uuid not found: 3ju5fLfCFyGNivksaq0z7YVGlpWhIX87yK0CoL4e9y86xtwpVDu8PS6WnSdfdlCU
  Failed to activate new LV.

I don't actually require access from that node to VG/LVs stored on that SAN. But I need to have that node synced in the cluster for resource management purposes.

Is there are a way I can tell LVM to ignore/exclude that node from clustered VG?


I ran into this a few years ago: As CLVM is directly tied to many parts of the cluster suite and was only designed to support coherent clusters, you cannot use it unless all of your machines see the same storage devices.

AFAICT there is no sane workaround.

  • The only "insanse" workaround I've found is to re-export that volume with other protocol. iSCSI for example. This a very dirt trick but it will work. – NStorm Feb 21 '18 at 12:48
  • @NStorm exactly what I meant but was afraid to write :) – Andreas Rogge Feb 21 '18 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.