0

I have two nodes , each has 20 HDDs (identical size) . Both nodes are configured with 2 VGs , each VG was created using 10 disks . 1 VG is called SAN the 2nd is called NAS .

Node1- will provide nfs services and the nfs data will be kept on the NAS vg , on an LV named nfs (it looks the same on node2) . VG SAN will be a backup of the SAN VG on node 2.

Node2- Will provide SAN services and the data will be kept on LVs . Each lv represents a lun and using targetcli , I'm creating block devices from each lv. So upon request for a lun , I create an LV with the needed size and create a backstore (blockdevice) in targetcli using this newly created LV.

VG NAS is also present on node2 and it's used to backup the data from node 1's NAS VG. The backed up data shouldn't be accessible to question : I thought about using drbd to backup the data from each node to the other so eventually each node 1 will backup node 2 and vice versa . With the NAS VG I'm ok , since there's only one lv on it and it's static. Question is , what happens with the SAN VG where I constantly create LVs? will drbd create them on the other side as well as a result of the backup ? if not , how can I keep those VGs identical at all time , meaning when I create an LV on node 2 it will automatically be created on node 1?

btw - the OS on both nodes is centos7.1

Thanks a lot for the help!

2 Answers 2

2

DRBD can use any block device as it's backing disk; LVM or the physical disk itself will work with DRBD. Also, DRBD creates a new virtual block device, which you will be able to use as a Physical Volume for LVMs if you wish. Any LV or VG created on DRBD can be activated on the peer node after DRBD has been promoted to Primary there.

You will need to tweak your LVM filter to pick up the LVM signatures on DRBD devices, and disable LVM filter caching in /etc/lvm/lvm.conf.

filter = ["a|/dev/sd.*|", "a|/dev/drbd.*|", "r|.*|"] 
write_cache_state = 0

After doing that, you should be able to use LVM "above and below" DRBD. Create the LVM to be used as DRBD's backing disk (example assumes sdb1 is the physical drive); perform these steps on both nodes:

# pvcreate /dev/sdb1
Physical volume "/dev/sdb1" successfully created
# vgcreate vg_drbd_back /dev/sdb1
Volume group "vg_drbd_back" successfully created
# lvcreate --name r0 --size 100G vg_drbd_back
Logical volume "r0" created

Create your DRBD configuration for r0 on both nodes:

resource r0 {
  device /dev/drbd0;
  disk /dev/vg_drbd_back/r0;
  meta-disk internal;
  on host-a { address x.x.x.x:7789; }
  on host-b { address x.x.x.x:7789; }
}

Initialize DRBD by creating the metadata and bringing the device up on both nodes, and then choose a single node to promote to Primary (--force only for the initial sync).

Then, on the Primary node, create the PV, VG, and LVM signatures on top of the DRBD device:

# pvcreate /dev/drbd0
Physical volume "/dev/drbd0" successfully created
# vgcreate vg_drbd_front /dev/drbd0
Volume group "vg_drbd_front" successfully created
# lvcreate --name drbd_lvm0 --size 30G vg_drbd_front
Logical volume "drbd_lvm0" created
# lvcreate --name drbd_lvm1 --size 40G vg_drbd_front
Logical volume "drbd_lvm1" created

To failover the LVM manually, you will need to deactivate the Volume Group on the Primary before you can demote it to Secondary:

# vgchange -a n vg_drbd_front
0 logical volume(s) in volume group "vg_drbd_front" now active
# drbdadm secondary r0

Then on the peer, promote DRBD and activate the Volume Group:

# drbdadm primary r0
# vgchange -a y vg_drbd_front
2 logical volume(s) in volume group "vg_drbd_front" now active

Of course, if you want to do this automatically with Pacemaker you will need to use the LVM resource agent which will handle the activation/deactivation of a specified VG:

# pcs resource describe ocf:heartbeat:LVM

Also, there are a lot of good (and free) technical guides and other resources at LINBIT's website that might be helpful in your clustering: http://www.linbit.com

EDIT: Added the section below regarding disabling 'lvmmetad'.

CentOS 7 enables the use of 'lvmetad' by default. You will need to disable that on both nodes for the filters to work as configured above.

In the /etc/lvm/lvm.conf set use_lvmetad = 0, and then stop lvm2-lvmmetad's service and socket via systemd:

# systemctl stop lvm2-lvmetad.service
# systemctl stop lvm2-lvmetad.socket
4
  • Thank you very much , @Matt . Perhaps my understanding of drbd is not the best , i'm still new to it ,but I want to make sure I understand what I'm doing.In the above example , we tweak drbd to pickup the lvm signatures . Then we create an LVM to be used as a backing disk for drbd, followed by creating drbd configurations . Why do we need the last part , i.e creating another lvm out of the drbd virtual device ? I kind of lost you there. Would be glad if you can explain the workflow . I thought you essentially create an LVM than drbd configurations and you're set.
    – John Doe
    Nov 16, 2015 at 12:18
  • You don't need to put LVM on top of DRBD, but if you wanted to, that's how you would do it. If you want to skip LVM on top, you won't need to change the LVM filters to scan DRBD devices. Nov 16, 2015 at 16:22
  • Hi @Matt , I've created (on both nodes) a new resource from all HDDs , configured one node as primary and from the drbd devices , created a pv-->vg-> lv but after the sync , the new lvm structure with the new vg and lv did not appear . I've tweaked the lvm conf file to pickup the signature as well. Any thoughts ?
    – John Doe
    Nov 22, 2015 at 9:03
  • @JohnDoe Funny you ask, I just ran into trouble with this on CentOS7 a few days ago. Anyway, I disabled 'lvmetad' to get it to work properly. I'll add an "UPDATE" section to the answer. Nov 23, 2015 at 19:33
0

You say backup, but drbd replicates immediately and does not count as a traditional offline backup. Its more like RAID 1. There still is a need to copy data you care about elsewhere, not vulnerable to accidental remove commands and the like.

drbd does copy everything on the block device including LVM and file system metadata. You need a clustered LVM to handle the devices on multiple nodes. If you do, yes the LVs can appear to either node. Red Hat supports this in as Pacemaker HA cluster resources. Their Logical Volume Manager Administration guide can get you started.

An alternative is to set up a distributed storage system like glusterfs. This doesn't use clvm but does allow distributed replicated volumes. This builds a certain type of shared file system on top of node local file system. clvm on drbd in contrast builds cluster file systems on top of replicated block disk devices.

1
  • Thank you for the reply . I actually configured pacemaker at first but I got the impression it will take care of the LV itself but not the vg , so if I'll create an LV on one node ,the other one will not auto create it . But now that you say that drbd will create it , that's exactly what I need. so it good. Do I configure drbd on the physical disks or the LVM ? (not sure if the LVM config will copy the metadata)
    – John Doe
    Nov 10, 2015 at 9:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .