In my case, the damage may have come from Windows, but other software could cause the same problem.
This definitely applies to ESXi 5.0 U3 and probably to any ESXi 5.x, and I’m guessing possibly any ESXi 6.x. It’s less likely to apply to ESXi 7, which uses a different, simpler partition layout.
Investigation Background
I was very close to doing a fresh install when I finally started figuring it out.
Poking around in /vmfs/volumes, one strange thing I noticed was that there were $RECYCLE.BIN directories in all of them, and these contained DESKTOP.INI files, with contents consistent with Windows Explorer shell extensions. This is a bit peculiar to find in the system partitions of a hypervisor that is not and never has been based on Windows.
I immediately became suspicious that the Windows OS I had used to copy in the disk image had done something to it. ESXi uses several FAT partitions on its boot disk, so perhaps Windows might play with them. Considering that the disk image was otherwise taken from a known-good state yet failed without doing anything at all within ESXi, this seemed the most promising course of investigation.
I was initially dismayed when I found, via a hex editor, that these $RECYCLE.BIN directories also appeared in the disk image. At first I took this to mean that the damage had already been done before the image was taken—also in Windows Server. However, these turned out to be benign, though they led me in the right direction. Windows likely added these as soon as it saw the original logical disk, which hadn’t been upsized yet—and in spite of the disk having been kept “Offline” the whole time.
Further poking around in the hex editor (HxD - Hex Editor and Disk Editor, really good tool) revealed one strange little difference between the GPT partition table in the disk image and the one live on the new RAID disk. This is not a difference a partition editor would have shown, because it actually makes no difference logically, as far as I know. This is something you’d have to be poking around in a raw hex dump to find.
Root Cause
In both cases, there were seven non-empty entries in the partition array. However, the way ESXi had originally written the array, there was a gap of one empty entry (all 128 bytes zeroed) after the first three entries, such that the last four valid entries fell into their own 512‑byte sector. On the live disk, where ESXi was borked, the last four valid entries had been shifted up one entry to close the gap. The entries were otherwise identical.
I don’t know who did this, Windows or HDD Raw Copy Tool, but I’d somewhat suspect Windows. I’ve tested it again and this change will be present immediately after the copy is done, even if the RAID logical drive is “Offline” in the Disk Management MMC snap‑in during and after the copy. It is an obviously intentional change, because the CRCs are all correct and the backup GPT is likewise changed.
My theory is that whoever is doing this is rewriting the GPT because it doesn’t match the size of the disk, and that instead of just copying the existing array exactly, it’s memorizing the partitions as a list in some internal structure and then re‑writing the array straightforwardly, which of course doesn’t produce a gap.
As a side note, the entries in the array as written by ESXi weren’t in the same order as the partitions occur physically on the disk, but whatever program closed the gap at least did not take the liberty of sorting the entries, too, thankfully!
Manual Fix
I don’t know of any easy way of recreating this gap, because generally any partition editor is going to do the same thing: convert the existing table to an internal representation used by the tool, do the changes you request on that representation, and then write the table back in GPT format such that it has the correct data. The exact on‑disk positioning of the array entries is as far as I know not supposed to be relevant and so won’t be part of the said “correct data” that it writes back.
However, I had a hunch that ESXi might be finicky about the exact array layout, so I decided to fix it manually to see what would happen. The procedure was:
- Make sure the disk is “Offline” in the Disk Management MMC snap‑in (as a precaution).
- Open the disk in a hex editor. In HxD it’s under Extras→Open disk... and you have to pick it from “Physical disks.” Be sure to uncheck “Open as Readonly,” which is on by default. You’ll get a suitable warning about this being dangerous.
- In the primary GPT’s array, which usually starts at LBA 2 (confirm at the header’s quadword at 48h), slide down the entries that are supposed to be after the gap by copying ranges, pasting further down, and then zeroing the gap. Better yet, just copy the actual table from the backup image, if you have one; but note that if the LUN’s size has changed you can’t just copy in the GPT header or else the situation is liable to repeat itself as that header will have the wrong values for fields 20h and 30h.
- Select the entire range of the GPT’s array. Technically you have to determine the range using the product of the doublewords at 50h and 54h in the GPT header, but this number will normally be 16,384 bytes.
- Take the CRC32 of the range selected in step 4. I wasn't able to find the mathematical parameters to the CRC32 algorithm in the UEFI specification, but was able to figure out it's a very common one, the one in ISO 802‑3, with the normal polynomial 04C11DB7h. You can compute it with an on‑line calculator here, being sure to set “Input type” to hex. Place this into the GPT header in little-endian at 58h.
- Temporarily place zero into the four bytes of the GPT header at 10h.
- Take the CRC32 of the header, whose length is specified in the header itself by the doubleword at 0Ch. Place this into the header at 10h.
- Repeat steps 3 through 7 for the backup GPT. The array and its CRC will be the same, so you can just copy these, but the header will be different and therefore have a different CRC. The header for the backup is normally at the very last sector of the disk, with the array immediately before it, but technically you should check quadword 20h in the primary GPT header and quadword 48h in the backup header to confirm.
- If you’re completely sure you’ve done all this right, hit Save. Again, HxD will give you a suitable warning prompt here.
You can consult the Wikipedia article for the basic technical details of the GPT format.
Screenshots
Here’s what part of the “broken” GPT array looked like; note that the valid entries end at 77Fh and that there are no long runs of zeros before then:
Here’s how I fixed it; note that the valid entries now end at 7FFh:
Outcome
I wasn’t really expecting this to work. I haven’t studied it but would expect that the UEFI specification attaches no significance to the ordering or spacing of array entries. In fact, every partition has a unique GUID precisely so that you don’t have to rely on such brittle heuristics. As such, depending on the array indices being the same as they were when you wrote the table would be a bad idea.
(Then again, it wouldn't be the first time I've seen enterprise-level software & firmware engage in bad ideas. Seems every other occasion on which I put on my sysadmin hat, I'm confronted with one or another blindingly stupid piece of programming... but let me not rant.)
So, I gingerly saved the tweaked table, booted back onto the RAID, ESXi started, vSphere Client connected and everything was back to normal.
Ideally, you’re usually better off using something like Veeam Backup, but in some situations more ad hoc solutions might be in order, in which case you could run into this bug.
