1

Before we start, yes i know the setup is not optimal, but we're working on a plan to setup everything from scratch. I inherited this from a part time student that doesn't work here anymore. This is a question on a potential solution.

We have a pretty urgent fault that needs to be fixed before the rebuilding of our NAS though.

The setup: 2x servers, 1x Windows server 2012r2 (referred to as Vmhost) and 1x FreeBSD (referred to as Nas) with 8x 300GB disks in Raidz2. The nas exposes 2 luns to vmhost with iscsi.

Vmhost is running a couple of VMs (duh) that is stored on these iscsi disks.

The problem: There is 1 VM that has snapshots (or checkpoints) in hyper-v that are 2 years old (don't ask why), which has resulted in that the snapshot file is just as big as the actual disk file (vhdx disk).

We are running of space on our nas and that makes our VMs slow or unresponsive.

One thing i do not understand (but is probably very easily explainable) is that Windows reports that the iscsi disk have almost 4TB of data on it, while i don't have more than 300*6 (+2 parity) GB of HDD installed. Is this just good compression on ZFS's part?

Screenshot of size in windows

Proposed solution: Move the vm's disk to some other disk and correct the mapping in vm config, then just press "delete checkpoint" in hyper-v and let hyper-v merge the snapshot to the disk and then move it back to the iscsi disk.

Question on is is: If the vhdx disk is 1tb and the snapshot file is 1tb, is a 3tb disk enough as a merge disk? And will it actually unlock 1tb? (the vhdx's size is fixed) or will it not do much (since it obviously is not reporting correct size)?

Bigger question: Since the sizes does not really make sense any direction, what numbers can i trust? I have way over 4tb data in total on 1800gb of space if i go by numbers. Is zfs so smart that is can see that some data in the snapshot might the same as on the disk and not use extra space?

zfs list

zpool list

hyper-v

windows explorer

1 Answer 1

1

In general, ZFS could easily achieve 1.2:1 compression of VM disk. It also is smart and doesn't save any zero blocks on disk. Databases compress better. In general, I wouldn't be surprised if 4Tb virtual disk of some good-compressible will fit into array with 1.8T physical space.

If you make snapshot with ZFS, it will be CoW snapshot. In the beginning, ZFS would know that data is the same and just store same data on disk once. When some copy gets a block written to, that block gets copied and stored in some other place (that's copy on write, CoW), then one of copies gets changed. From on now, even if you revert copies to have same data, they wouldn't be merged in general case.

You can merge any blocks that happened to be the same, even if you not beginning with snapshot. You can tell it to check if the data area has some duplicate blocks and, you guess it, store them only once. This feature called deduplication, but I don't see you are using it, because you have DEDUP 1.00x in zpool list. Also, beware, this feature consumes a LOT of memory.

I'd begin your case with booting your suspicious Windows VM some disk backup software which does backups by taking sector-by-sector copies and do a complete backup of virtual disk from within of that VM (i.e. backup what Windows sees). Or/and backup virtual disk images. That will allow to at least not to lose data you certainly can access. Then act as you described.

Also a side note. I see something called "virtual backup" on your screenshot. Don't do backups like this. Don't rely on same logical structure. You'll lose that backup together with data it has to protect if some rare error in your stack (NTFS in virtual disk over NTFS on base disk over ZFS pool) destroys on-disk structures, and it is impossible to do forensic recovery of data from such double cake. It is often best to store backups into a simplest possible structure which is independent from main storage, i.e. a separate disk, just bare partition table without advanced volume management, simplest file system. That's easier to access and recover in any case.

2
  • Ahh thanks a lot for the detailed explanation! And the 1TB vm disk is only about 1/3 full, so ye if it doesn't store zeros it makes a lot more sense. And the virtual backup server is just a small ubuntu machine that backs up our databases on AWS, Github repos and some other things in the cloud to have a local copy just in case. We do however have a 1 other bare metal server that we use zfs send / receive to backup snapshot from zfsnap, and a vm on AWS that does the same (for onsite and offsite backup) Oct 10, 2019 at 18:44
  • And yes, my plan was to copy the vhdx over to 2 different places. 1 place to have a backup incase i (or hyper-v) screws up and 1 where i will do the hyper-v merge. I was planning on just copying with windows explorer though. Oct 10, 2019 at 18:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .