First off I know this topic easily gets subjective, but I'm trying to avoid that, as there should be at least 1 good answer to this in a sea of bad answers and it's hard to find.
At first sight my question seems simple; How do you store Virtual Machine disks on Harddisks, while making sure data integrity isn't compromised, and performance isn't horrible.
But it's actually harder than it appears;
- ZFS and BTRFS are no option: Copy On Write filesystems are notoriously poor at handling large files, especially if they may contain another Copy On Write filesystem themselves! You CAN turn off COW on BTRFS, but this also turns off checksumming (and compression, deduplication, etc).
- EXT2/3/4, XFS, ReiserFS, NTFS, etc etc all do not do full data checksumming like BTRFS/ZFS do and are not and option.
So is it checkmate then? You can't have full data integrity other than simplistic RAID setups that have other issues such as Write Holes(RAID5) and generally very poor handling of corrupt files where it's unclear which of two copies is the correct one. Issues that are avoided with higher level systems that checksum and verify the integrity of files before they're returned to the operating system or user.
The only option I can think of is using BTRFS/ZFS inside the VMs rather than on the host, and schedule snapshots and backups appropriately on each machine, even though that's a lot more cumbersome than doing it on the host.
Does anyone know any other way to achieve my goal?