0

I have a server which runs Xen 4.9.2 with Linux Ubuntu 18.04.2 installed in Dom0. There is one virtual machine (DomU) with Linux Ubuntu 12.04.5 that uses LVM logical volume as its root filesystem. This LVM logical volume is configured in Dom0 and ext4 filesystem is created on it. Then it is exposed to DomU as follows:

disk = [ '/dev/GuestVg/RootLv0,raw,xvda1,rw' ]

and DomU mounts it this way (/etc/fstab):

# file system  mount point  type  options                              dump    pass
/dev/xvda1     /            ext4  noatime,nodiratime,errors=remount-ro 0       1

I am trying to use LVM snapshot to backup the logical volume from Dom0 while the filesystem is mounted in DomU. The problem with this approach is that when I create read-only snapshot of this logical volume, it gets created with filesystem in a "dirty" state (needs_recovery flag is set) and my backup tool fsarchiver cannot mount it.

# Creating read-only snapshot
sudo lvcreate -s -p r -n rootfs_snapshot -L 512M /dev/GuestVg/RootLv0
  Using default stripesize 64.00 KiB.
  Logical volume "rootfs_snapshot" created.

# Checking needs_recovery flag on the filesystem
sudo tune2fs -l /dev/GuestVg/RootLv0 | grep 'needs_recovery'
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent sparse_super large_file

# Checking needs_recovery flag on the snapshot
sudo tune2fs -l /dev/GuestVg/rootfs_snapshot | grep 'needs_recovery'
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent sparse_super large_file

# Running backup
sudo fsarchiver -v savefs rootfs_2019-02-15.fsa /dev/GuestVg/rootfs_snapshot
filesys.c#318,generic_mount(): partition [/dev/GuestVg/rootfs_snapshot] cannot be mounted on [/tmp/fsa/20190215-205030-00061846-00] as [ext4] with options [user_xattr,acl]
oper_save.c#1032,filesystem_mount_partition(): cannot mount partition [/dev/GuestVg/rootfs_snapshot]: filesystem may not be supported by either fsarchiver or the kernel.
removed rootfs_2019-02-15.fsa

This way of making backup is described in a lot of manuals and guidelines on the internet and noone seems to notice the problem that I am facing.

When the filesystem is unmounted or when it is mounted in Dom0 (instead of DomU), the snapshot gets created without needs_recovery flag, and it can be successfully mounted and backed up.

I did some research and found an explanation of what happens in case when the filesystem is mounted in Dom0. Reference: https://yarchive.net/comp/linux/disk_snapshot.html

When ext4 filesystem gets mounted, needs_recovery flag is set. This flag is cleaned during clean unmount of the filesystem. If the system crashes without clean umount this flag lets e2fsck program know that the check needs to be performed on the filesystem to recover it. When LVM snapshot is taken of a logical volume with mounted ext4 filesystem, lvcreate command takes the snapshot of the filesystem in a quiescent state; that is, it will force the journal transaction to close, suspend all filesystem activity, take a snapshot of the disk as if it had been unmounted, and then allow filesystem activity to continue. So if you look at an ext4 filesystem snapshot taken in this way, you will see that the needs_recovery flag is not set, since the ext4 journal is empty in the snapshot.

My problem is that the filesystem is mounted by the kernel running in DomU, not Dom0 (where the snapshot is taken) and therefore all described steps that lvcreate takes seem to have no effect. The snapshot is taken of the filesystem in a "dirty" state (needs_recovery flag is still set) and tools like fsarchiver refuse to mount such snapshot to take backup. I can make some changes to the filesystem via tune2fs from Dom0 to clean needs_recovery flag (while it is still mounted in DomU) and then take the snapshot but it feels like a hack that may result in data corruption because kernel in DomU will not know that the filesystem was modified from Dom0.

So my question is: what is the correct way to take LVM snapshot in a clean state in Dom0 of a logical volume with a filesystem mounted in running DomU? Is it even possible?

I would not like to shutdown DomU to make a backup but I wonder if this is the only way to make a consistent snapshot.

  • Not completely sure how you would do it, but for ESX for example a quiescent snapshot will trigger Open-vm-tools vmbackup.start on the guest to prepare the snapshot (and run scripts in a Dir as well as legacy /usr/sbin/pre-freeze-script and post-thaw-Script hooks.) – eckes Feb 16 at 18:09
  • The qemu guest agent also has freeze and unfreeeze options wiki.qemu.org/Features/GuestAgent funny enough for XenServer quiescent snapshots are only specific for Windows VSS guests, no idea why that it. Xen seems to ignore Linux guests in that regard. – eckes Feb 16 at 18:26
0

Out of curiosity, how are you doing the backups? If you're just creating a snapshot, then using something like rsync to copy the contents out to a backup folder, then would it matter if you mount the snapshot in read-only mode, or if the mount is "dirty"?

The reason I'm asking is because I'm doing the same thing, and it works fine. When you mount the snapshot read-write then do a lvremove, there isn't a risk of changes being written back to the base volume. Using rsync to copy the file contents out of the mounted snapshot isn't any different than if you did the backup from the guest itself -- databases/files will still be open, etc., so you need to act accordingly and exclude them from the backup (and backup them separately.)

I could see this being an issue if you were doing an image-based backup of the snapshot volume (since the image would be in an inconsistent state,) but doing a file-based backup shouldn't be an issue as long as you handle it appropriately.

  • I create read-only LVM snapshot and do file-based backup using fsarchiver tool. The tool performs the mount of the snapshot itself and fails to mount a snapshot in dirty state. To answer your second question. When a mounted file system is getting random modifications all the time, taking a snapshot without freezing the I/O may result in corrupted data. I do not want to discover the data corruption when I need to restore the data from the snapshot later. – Oleg Lypkan Feb 22 at 5:00
  • I did some more research and found that there is a command fsfreeze that puts the filesystem in a consistent state by flushing caches/buffers, cleans journal, etc and halts all new write operations on the filesystem until it is unfrozen. – Oleg Lypkan Feb 22 at 5:01
  • Similar thing is done by lvcreate when it creates a snapshot of a local filesystem. However, lvcreate cannot perform fsfreeze operation on a filesystem mounted in a guest OS. I tried running fsfreeze via ssh on the guest OS but then the whole session hanged (probably because the shell/ssh server tries to perform write operation to the frozen filesystem) and I could not ssh again. So, I need to play more with it to find a save way freeze/unfreeze the filesystem. – Oleg Lypkan Feb 22 at 5:01
  • Based on linux.die.net/man/8/fsfreeze, it sounds like running fsfreeze will in effect pause all access to the filesystem, not just cause it to flush buffers and allow it to resume later. So I'm guessing that you wouldn't want to run that on your root volume? – smily03 Feb 23 at 2:46
  • It is true that a mounted filesystem will have continuous modifications, but if you do any sort of file-based copy from the guest OS directly, the same caveat would be there as well. Instead of issuing something like fsfreeze, could you just execute the "sync" command on the guest OS (via SSH?) immediately before taking the snapshot? – smily03 Feb 23 at 2:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.