Suppose we have a datastore on a local disk, and thereʼs a VM that has a virtual HDD whose
.vmdk file resides in that datastore.
Normally, if the VM tries to read the disk, for instance, it is actually reading blocks in the
.vmdk file. Those blocks in turn have to be read from the local disk the datastore is on. So, the data goes from the physical disk to the VMware kernelʼs VMFS system, then to the hypervisor which then emulates a SCSI controller which the guest OS reads from.
But what happens if the physical disk says, “oops, sorry, that sector is bad.”
Well, if the sector happened to be part of the VMFS metadata, then what happens would depend on the resiliency of VMFS. However, in this case, letʼs suppose that the bad sector is in user data, from the perspective of VMFS. In other words, VMFS is structurally intact, but the data for a particular block in the
.vmdk file is lost.
Now what does the guest OS see?
There are two possibilities I see. The first possibility, and the way it would work had I engineered it, would be that VMware would actually tell the guest that the read didnʼt work. That is, it would make the emulated SCSI controller do whatever a SCSI controller does when the disk barfs on a sector.
Unfortunately, in my experience, software often doesnʼt take the logical route. So, itʼs also possible that VMware would just silently rewrite zeros into the bad sector to get the disk to remap it, then return a sector full of zeros to the guest as though nothing had gone wrong.
Thatʼd be pretty stupid, but I have to wonder. The reason I wonder is that Iʼve got such disks in a ZFS raidz in a Solaris guest, and after resilvering, I found one of my drives in the guest showed:
NAME STATE READ WRITE CKSUM ... c7t4d0 ONLINE 0 0 19
These errors were fixed with parity data from the other drives, so the pool has no data errors, but I still want to know what exactly happened.
As best as I understand the above report, it means that
c7t4d0 obligingly returned a sector full of data, but the data was wrong as determined by cryptographic hash. This happened not once but 19 times! I have just enough faith in the hamming codes used by HDD manufacturers to suspect that, while you might see a silent corruption once in a dogʼs age, getting 19 of them all at once like this is astronomically unlikely.
It seems much more likely to me that the corruption actually wasnʼt silent — that the HDD actually told VMware that the sectors couldnʼt be read, but that VMware went off and just filled them with zeros or something, which would of course result in checksum errors at the guest level. I would consider this a rather onerous bug! The harm couldʼve been quite serious had this not been a filesystem like ZFS thatʼs smart enough to pick up on trashed user data.
So, my question is, what exactly does VMware do in these situations where the backing physical disk posts I/O errors? The above evidence seems to hint at it doing the wrong thing, but I donʼt really know.
I realize youʼd typically use SCSI passthrough in this particular application and that youʼd always get the expected result in that case, but for reasons I canʼt get into here, I couldnʼt use passthrough in this case. And anyway, Iʼd still expect VMware to do the right thing.
This particular system is on an older VMware, ESXi 5, but Iʼm interested in how this is handled in general, including newer versions.