How to recover a cached LVM with cache on ram-disk?

Question

In this article there is a nice recipe for how to use a RAM-disk as cache-device for a classical LVM volume.

Assumed you have an elder disk, lots of RAM and no SSD, you can boost disk performance to native RAM-throughput using this technique.

So I did this on an LVM which is used for my virtual VM running Windows 10. Voila: Disk-throughput was 4 times faster within the VM (average throughput, best used while patching Windows).

All was well - until I shutdown my linux-system (CentOS 7).

Data Loss!

The shutdown will not disassemble that cache. The same would be true in a power-failure-situation (yes - there will be data loss).

However - there has to be a way to recover what is left. But LVM will not let you operate on a VG with missing disks.

So - is there a receipe for this case out there?

Like

• recover missing LVM cache disk with a new disk
• force clean state
• access cached LV again

In the last step one would make filesystem repairs and would recover missing/corrupted files from backup (using rsync).

Answer 1

you can boost disk performance to native RAM-throughput using this technique

No, not quite. Once the cache is full of writes, write throughput degrades to what the underlying disk can do. And the first reads from the backing disk are still slow. I/O to the cache, yes is faster.

---

This method is very risky because it is very much a not persistent disk. I'm skeptical of the utility when you can just get a fast solid state disk or hardware write cache, but failure modes are fun to test.

THIS WILL CAUSE DATA LOSS. ONLY PROCEED IF YOU HAVE A BACKUP OF YOUR DATA.

First, without cache.

# Create volume
pvcreate /dev/sdb
vgcreate vg /dev/sdb
lvcreate --size 400g --name lv vg
mkfs.xfs /dev/vg/lv
mkdir /mnt/lv
mount /dev/vg/lv /mnt/lv
# Write test
dd bs=1M if=/dev/zero of=/mnt/lv/zero count=10000

Add cache.

# Create a RAM disk
modprobe brd rd_nr=1 rd_size=4585760 max_part=0
pvcreate /dev/ram0
vgextend vg /dev/ram0
# Create a cache
lvcreate -L 300M -n cache_meta vg /dev/ram0
lvcreate -L 4G -n cache_vol vg /dev/ram0
lvconvert –type cache-pool –poolmetadata vg/cache_meta –cachemode=writeback vg/cache_vol -y
# Add cache to a LV
lvconvert –type cache –cachepool vg/cache_vol vg/lv
# Write test
dd bs=1M if=/dev/zero of=/mnt/lv/zero2 count=10000
# Crash test
echo 'c' > /proc/sysrq-trigger

When it comes back, LVM is very unhappy, the volume is unaccessible.

[root@sf ~]# lvs
  WARNING: Device for PV YpvOB5-PZLO-POFL-3Cf4-G1IB-gep8-6eU10R not found or rejected by a filter.
  LV   VG Attr       LSize   Pool        Origin     Data%  Meta%  Move Log Cpy%Sync Convert
  lv   vg Cwi---C-p- 400.00g [cache_vol] [lv_corig]
[root@sf ~]# mount /dev/vg/lv /mnt/lv/
mount: special device /dev/vg/lv does not exist
[root@sf ~]# pvs
  WARNING: Device for PV YpvOB5-PZLO-POFL-3Cf4-G1IB-gep8-6eU10R not found or rejected by a filter.
  PV         VG Fmt  Attr PSize    PFree
  /dev/sdb   vg lvm2 a--  <500.00g 99.70g
  [unknown]  vg lvm2 a-m     4.37g 80.00m

You can't even force uncache it because the metadata has an I/O error.

[root@sf ~]# lvconvert --uncache vg/lv --force -y
  WARNING: Device for PV YpvOB5-PZLO-POFL-3Cf4-G1IB-gep8-6eU10R not found or rejected by a filter.
  WARNING: Cache pool data logical volume vg/cache_vol_cdata is missing.
  WARNING: Cache pool metadata logical volume vg/cache_vol_cmeta is missing.
  WARNING: Uncaching of partially missing writethrough cache volume vg/lv might destroy your data.
  /dev/mapper/vg-cache_vol_cmeta: read failed: Input/output error
  Failed to active cache locally vg/lv.

But you can force the data loss by creating a new PV with the same UUID. Uncache it to remove the ramdisk PV that LVM thinks still has data in it, but was lost. You could re-add the new cache with lvconvert, but I am not going to after the results of this experiment.

pvcreate --norestore --uuid YpvOB5-PZLO-POFL-3Cf4-G1IB-gep8-6eU10R  /dev/ram0
lvconvert --uncache vg/lv

Finally, check for any file system damage. Restore from backup is required to get your data back into a good state.

xfs_repair /dev/vg/lv

Edit: adding an empty PV in with the same UUID back in seems super hacky. lvconvert refused to uncache it, after all. If instead you put the metadata LV on permanent disk, then it can be cleaned up a little easier.

# Same procedure but meta is on persistent storage.
lvcreate -L 300M -n cache_meta vg /dev/sdb

It can be forced to uncache. Don't let the "Flushing 0 blocks" output reassure you, in-flight writes were already lost. The missing ramdisk can then be removed, making the VG consistent again.

[root@sf ~]# lvconvert --uncache vg/lv --force -y
  WARNING: Device for PV YpvOB5-PZLO-POFL-3Cf4-G1IB-gep8-6eU10R not found or rejected by a filter.
  WARNING: Cache pool data logical volume vg/cache_vol_cdata is missing.
  WARNING: Uncaching of partially missing writethrough cache volume vg/lv might destroy your data.
  Flushing 0 blocks for cache vg/lv.
  Logical volume "cache_vol" successfully removed
  Logical volume vg/lv is not cached.
[root@sf ~]# vgreduce --removemissing vg
  WARNING: Device for PV YpvOB5-PZLO-POFL-3Cf4-G1IB-gep8-6eU10R not found or rejected by a filter.
  Wrote out consistent volume group vg.
[root@sf ~]# pvs
  PV         VG Fmt  Attr PSize    PFree
  /dev/sdb   vg lvm2 a--  <500.00g <100.00g

Answer 2

I tried to follow the other answer. However, I did not set up the ram to be the cache disk but another SSD and the SSD broke. Thus, the answer did not work for me since I was not able to create another physical volume since I did not have the same SSD twice.

So I came up with another solution which should work for a wider range of szenarios:

• Everything is as it is in the other answer
• But some more adjustments are necessary when creating the new dummy pv since our original pv is completely gone

# check again for available lvs and pvs
lvs -a
pvs -a

# create config backup of the vg with the missing cache pv
vgcfgbackup -f vgdata_backup.vg vg-data

# create a second backup
cp vgdata_backup.vg vgdata_new.vg

# open the config
nano vgdata_new.vg

# search for the size of your missing pv, the 512 is the block size, which can remain like this
truncate -s $((216825856 * 512)) test.img

# create a loopback device (virtual disk) from the created file
losetup /dev/loop101 ./test.img

# now create a pv from the new disk and insert your uuid which you can also find in the config backup
pvcreate --norestore --uuid VJW345-hle6-0yqY-Pgnm-ldWL-invq-OW4Vvc /dev/loop101

# now edit the config
# you have to adjust the path to your file, and probably the flags, like this
pv2 {
        id = "VJW345-hle6-0yqY-Pgnm-ldWL-invq-OW4Vvc"
        device = "/dev/loop101" # Hint only

        status = ["ALLOCATABLE"]
        flags = []
        dev_size = 216825856    # 103,391 Gigabytes
        pe_start = 2048
        pe_count = 26467        # 103,387 Gigabytes
}

# now restore the new config
vgcfgrestore -f vgdata_new.vg vg-data

# check again
lvs -a
pvs -a

# FINALLY: you made it and you can uncache the vg
lvconvert --uncache vg-data/lv-0 --force