0

I’m writing this after having found the solution myself, because it was such a strange “gotcha” it deserves to be documented.

Although I hit this issue while doing a restore, it’s possible this could come up in other cases of booting into another operating system while the boot drive of an ESXi installation is attached to the system, particularly if the disk size has changed.

I just recently restored a VMware ESXi installation’s boot drive, also containing a datastore holding most of the VMs and their virtual boot/system disks, only to find that this supposedly known-good state was somehow borked.

Based on the display on the server’s local console, ESXi appeared to be booting normally, but it was exhibiting many problematic symptoms:

  • Couldn’t log into it with vSphere Client, which gave the message, “vSphere Client could not connect to host. An unknown connection error occurred. (The request failed because of a connection failure. (Unable to connect to the remote server))”.

  • vSphere Client’s log included an error:

    System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it

  • Attempting to log into the server with a web browser, the browser reported the connection was refused.

  • Could not SSH into the server even after enabling this on the local console.

  • Restarting the management network and the management agents via the local console did not fix it.

  • On the local console, discovered hostd was not running, and rebooting didn’t fix it.

  • esxcli commands always gave: Connect to localhost failed: Connection failure

  • There were less directories under /vmfs/volumes than expected and none of them appeared to be any of my datastores.

  • Even “Reset System Configuration” in the local UI didn’t fix it. (I only tried this because I had a known-good image I could restore from again, which I did after solving the problem, though I’m not sure it had even changed anything.)

The backup copy I was restoring from was a low-level image of the RAID logical disk taken with the server down. After deleting a potentially corrupted RAID array and recreating it, I used a separate drive attached on an HBA to boot into a physical Windows Server installation to copy the image onto the new RAID. I used HDD Raw Copy Tool from hddguru.com, which is basically a less cryptic and nail-biting alternative to the Linux dd command.

(This is an admittedly barbaric—albeit quite complete—way of backing up VMware, but this disk mainly only stores VM boot/system disks, not data, so it’s not backed up very frequently, anyway, except when doing major changes. We have much better backup systems for primary data.)

I made the new RAID logical disk bigger than the one that had been backed up, since we’ve upgraded to larger drives in the RAID and the datastore was a little full. I had been planning to grow the datastore, after confirming the backup worked.

No sooner did I get the raw copy done than I booted into ESXi to find it borked. What happened?!

This is a pretty old ESXi 5.0 U3. (It’s been meeting our present needs fine and we don’t have any full-time IT staff to manage upgrades for the sake of upgrades, fix the problems they often cause, etc.)

0

In my case, the damage may have come from Windows, but other software could cause the same problem.

This definitely applies to ESXi 5.0 U3 and probably to any ESXi 5.x, and I’m guessing possibly any ESXi 6.x. It’s less likely to apply to ESXi 7, which uses a different, simpler partition layout.

Investigation Background

I was very close to doing a fresh install when I finally started figuring it out.

Poking around in /vmfs/volumes, one strange thing I noticed was that there were $RECYCLE.BIN directories in all of them, and these contained DESKTOP.INI files, with contents consistent with Windows Explorer shell extensions. This is a bit peculiar to find in the system partitions of a hypervisor that is not and never has been based on Windows.

I immediately became suspicious that the Windows OS I had used to copy in the disk image had done something to it. ESXi uses several FAT partitions on its boot disk, so perhaps Windows might play with them. Considering that the disk image was otherwise taken from a known-good state yet failed without doing anything at all within ESXi, this seemed the most promising course of investigation.

I was initially dismayed when I found, via a hex editor, that these $RECYCLE.BIN directories also appeared in the disk image. At first I took this to mean that the damage had already been done before the image was taken—also in Windows Server. However, these turned out to be benign, though they led me in the right direction. Windows likely added these as soon as it saw the original logical disk, which hadn’t been upsized yet—and in spite of the disk having been kept “Offline” the whole time.

Further poking around in the hex editor (HxD - Hex Editor and Disk Editor, really good tool) revealed one strange little difference between the GPT partition table in the disk image and the one live on the new RAID disk. This is not a difference a partition editor would have shown, because it actually makes no difference logically, as far as I know. This is something you’d have to be poking around in a raw hex dump to find.

Root Cause

In both cases, there were seven non-empty entries in the partition array. However, the way ESXi had originally written the array, there was a gap of one empty entry (all 128 bytes zeroed) after the first three entries, such that the last four valid entries fell into their own 512‑byte sector. On the live disk, where ESXi was borked, the last four valid entries had been shifted up one entry to close the gap. The entries were otherwise identical.

I don’t know who did this, Windows or HDD Raw Copy Tool, but I’d somewhat suspect Windows. I’ve tested it again and this change will be present immediately after the copy is done, even if the RAID logical drive is “Offline” in the Disk Management MMC snap‑in during and after the copy. It is an obviously intentional change, because the CRCs are all correct and the backup GPT is likewise changed.

My theory is that whoever is doing this is rewriting the GPT because it doesn’t match the size of the disk, and that instead of just copying the existing array exactly, it’s memorizing the partitions as a list in some internal structure and then re‑writing the array straightforwardly, which of course doesn’t produce a gap.

As a side note, the entries in the array as written by ESXi weren’t in the same order as the partitions occur physically on the disk, but whatever program closed the gap at least did not take the liberty of sorting the entries, too, thankfully!

Manual Fix

I don’t know of any easy way of recreating this gap, because generally any partition editor is going to do the same thing: convert the existing table to an internal representation used by the tool, do the changes you request on that representation, and then write the table back in GPT format such that it has the correct data. The exact on‑disk positioning of the array entries is as far as I know not supposed to be relevant and so won’t be part of the said “correct data” that it writes back.

However, I had a hunch that ESXi might be finicky about the exact array layout, so I decided to fix it manually to see what would happen. The procedure was:

  1. Make sure the disk is “Offline” in the Disk Management MMC snap‑in (as a precaution).
  2. Open the disk in a hex editor. In HxD it’s under ExtrasOpen disk... and you have to pick it from “Physical disks.” Be sure to uncheck “Open as Readonly,” which is on by default. You’ll get a suitable warning about this being dangerous.
  3. In the primary GPT’s array, which usually starts at LBA 2 (confirm at the header’s quadword at 48h), slide down the entries that are supposed to be after the gap by copying ranges, pasting further down, and then zeroing the gap. Better yet, just copy the actual table from the backup image, if you have one; but note that if the LUN’s size has changed you can’t just copy in the GPT header or else the situation is liable to repeat itself as that header will have the wrong values for fields 20h and 30h.
  4. Select the entire range of the GPT’s array. Technically you have to determine the range using the product of the doublewords at 50h and 54h in the GPT header, but this number will normally be 16,384 bytes.
  5. Take the CRC32 of the range selected in step 4. I wasn't able to find the mathematical parameters to the CRC32 algorithm in the UEFI specification, but was able to figure out it's a very common one, the one in ISO 802‑3, with the normal polynomial 04C11DB7h. You can compute it with an on‑line calculator here, being sure to set “Input type” to hex. Place this into the GPT header in little-endian at 58h.
  6. Temporarily place zero into the four bytes of the GPT header at 10h.
  7. Take the CRC32 of the header, whose length is specified in the header itself by the doubleword at 0Ch. Place this into the header at 10h.
  8. Repeat steps 3 through 7 for the backup GPT. The array and its CRC will be the same, so you can just copy these, but the header will be different and therefore have a different CRC. The header for the backup is normally at the very last sector of the disk, with the array immediately before it, but technically you should check quadword 20h in the primary GPT header and quadword 48h in the backup header to confirm.
  9. If you’re completely sure you’ve done all this right, hit Save. Again, HxD will give you a suitable warning prompt here.

You can consult the Wikipedia article for the basic technical details of the GPT format.

Screenshots

Here’s what part of the “broken” GPT array looked like; note that the valid entries end at 77Fh and that there are no long runs of zeros before then:

hex dump of broken GPT, without gap

Here’s how I fixed it; note that the valid entries now end at 7FFh:

hex dump of fixed GPT, with gap

Outcome

I wasn’t really expecting this to work. I haven’t studied it but would expect that the UEFI specification attaches no significance to the ordering or spacing of array entries. In fact, every partition has a unique GUID precisely so that you don’t have to rely on such brittle heuristics. As such, depending on the array indices being the same as they were when you wrote the table would be a bad idea.

(Then again, it wouldn't be the first time I've seen enterprise-level software & firmware engage in bad ideas. Seems every other occasion on which I put on my sysadmin hat, I'm confronted with one or another blindingly stupid piece of programming... but let me not rant.)

So, I gingerly saved the tweaked table, booted back onto the RAID, ESXi started, vSphere Client connected and everything was back to normal.

Ideally, you’re usually better off using something like Veeam Backup, but in some situations more ad hoc solutions might be in order, in which case you could run into this bug.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.