I have a server which runs Xen 4.9.2 with Linux Ubuntu 18.04.2 installed in Dom0. There is one virtual machine (DomU) with Linux Ubuntu 12.04.5 that uses LVM logical volume as its root filesystem. This LVM logical volume is configured in Dom0 and ext4 filesystem is created on it. Then it is exposed to DomU as follows:
disk = [ '/dev/GuestVg/RootLv0,raw,xvda1,rw' ]
and DomU mounts it this way (/etc/fstab):
# file system mount point type options dump pass /dev/xvda1 / ext4 noatime,nodiratime,errors=remount-ro 0 1
I am trying to use LVM snapshot to backup the logical volume from Dom0 while the filesystem is mounted in DomU. The problem with this approach is that when I create read-only snapshot of this logical volume, it gets created with filesystem in a "dirty" state (needs_recovery flag is set) and my backup tool fsarchiver cannot mount it.
# Creating read-only snapshot sudo lvcreate -s -p r -n rootfs_snapshot -L 512M /dev/GuestVg/RootLv0 Using default stripesize 64.00 KiB. Logical volume "rootfs_snapshot" created. # Checking needs_recovery flag on the filesystem sudo tune2fs -l /dev/GuestVg/RootLv0 | grep 'needs_recovery' Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent sparse_super large_file # Checking needs_recovery flag on the snapshot sudo tune2fs -l /dev/GuestVg/rootfs_snapshot | grep 'needs_recovery' Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent sparse_super large_file # Running backup sudo fsarchiver -v savefs rootfs_2019-02-15.fsa /dev/GuestVg/rootfs_snapshot filesys.c#318,generic_mount(): partition [/dev/GuestVg/rootfs_snapshot] cannot be mounted on [/tmp/fsa/20190215-205030-00061846-00] as [ext4] with options [user_xattr,acl] oper_save.c#1032,filesystem_mount_partition(): cannot mount partition [/dev/GuestVg/rootfs_snapshot]: filesystem may not be supported by either fsarchiver or the kernel. removed rootfs_2019-02-15.fsa
This way of making backup is described in a lot of manuals and guidelines on the internet and noone seems to notice the problem that I am facing.
When the filesystem is unmounted or when it is mounted in Dom0 (instead of DomU), the snapshot gets created without needs_recovery flag, and it can be successfully mounted and backed up.
I did some research and found an explanation of what happens in case when the filesystem is mounted in Dom0. Reference: https://yarchive.net/comp/linux/disk_snapshot.html
When ext4 filesystem gets mounted, needs_recovery flag is set. This flag is cleaned during clean unmount of the filesystem. If the system crashes without clean umount this flag lets e2fsck program know that the check needs to be performed on the filesystem to recover it. When LVM snapshot is taken of a logical volume with mounted ext4 filesystem, lvcreate command takes the snapshot of the filesystem in a quiescent state; that is, it will force the journal transaction to close, suspend all filesystem activity, take a snapshot of the disk as if it had been unmounted, and then allow filesystem activity to continue. So if you look at an ext4 filesystem snapshot taken in this way, you will see that the needs_recovery flag is not set, since the ext4 journal is empty in the snapshot.
My problem is that the filesystem is mounted by the kernel running in DomU, not Dom0 (where the snapshot is taken) and therefore all described steps that lvcreate takes seem to have no effect. The snapshot is taken of the filesystem in a "dirty" state (needs_recovery flag is still set) and tools like fsarchiver refuse to mount such snapshot to take backup. I can make some changes to the filesystem via tune2fs from Dom0 to clean needs_recovery flag (while it is still mounted in DomU) and then take the snapshot but it feels like a hack that may result in data corruption because kernel in DomU will not know that the filesystem was modified from Dom0.
So my question is: what is the correct way to take LVM snapshot in a clean state in Dom0 of a logical volume with a filesystem mounted in running DomU? Is it even possible?
I would not like to shutdown DomU to make a backup but I wonder if this is the only way to make a consistent snapshot.