CLVM with shared storage not available to all nodes in the cluster

Question

If you are running a 3 or more node cluster with cman/corosync and you have some sort of shared block storage (SAN) connected only to some nodes in the cluster how can you enable CLVM on that storage device?

In my example I'm running a 3 node cluster, there 2 nodes are "main workload" and 1 node are used for backups & archives. Main nodes are connected with FC HBA with multipath to a SAN. Everything works fine, I was able to initialize PV on that device and can see it on both nodes:

  PV                 VG      Fmt  Attr PSize   PFree  
  /dev/mapper/mpathd san-vg1 lvm2 a--  500.00g 500.00g

Now, to create a clustered VG, I've enabled fencing on all 3 nodes in my cluster (with IPMI if that matters).

fence domain
member count  3
victim count  0
victim now    0
master nodeid 1
wait state    none
members       1 2 3 

I've also created a clustered VG (vgcreate -c)on that PV and that one works fine too:

  VG      #PV #LV #SN Attr   VSize   VFree  
  san-vg1   1   0   0 wz--nc 500.00g 500.00g

But I cannot create LV on that VG now, because 3rd node in the cluster doesn't have FC HBA, and don't "see" that PV at all:

  Error locking on node main-stor1: Volume group for uuid not found: 3ju5fLfCFyGNivksaq0z7YVGlpWhIX87yK0CoL4e9y86xtwpVDu8PS6WnSdfdlCU
  Failed to activate new LV.

I don't actually require access from that node to VG/LVs stored on that SAN. But I need to have that node synced in the cluster for resource management purposes.

Is there are a way I can tell LVM to ignore/exclude that node from clustered VG?

Answer 1

I ran into this a few years ago: As CLVM is directly tied to many parts of the cluster suite and was only designed to support coherent clusters, you cannot use it unless all of your machines see the same storage devices.

AFAICT there is no sane workaround.