Here's the process I used to resolve this:
VMware ESXi can be installed in an embedded mode or an installable mode. As outlined here, the installation mode is determined by the destination media and the size of the volume available to the ESXi installer.
- USB, SDHC or any device less than 5GB in size: Embedded
- Hard drives/volumes greater than or equal to 5GB in size: Installable
One of the unique attributes of running ESXi in embedded mode is that the OS is loaded into RAM and only touches the USB/SD device hourly during normal operation. In my situation, the system continued to operate, even with a failed SDHC device.
The error message I received in the vCenter interface indicated that configuration changes would not be saved, but the cluster was still usable. I left the system in this state for several days until I could get to the datacenter to replace the SD card.
With regard to steps to take following a failure of a USB or SD device, it is important to extract and save a copy of your host's settings!!
This is easily accomplished via PowerCLI or the vSphere CLI. I used PowerCLI running from the vCenter server:
Get-VMHostFirmware -VMHost 10.10.8.22 -BackupConfiguration -DestinationPath C:\Users\ewwhite\Downloads
- Following that, I evacuated all virtual machines from the affected host and placed it in maintenance mode.
- The host was then shut down, the SDHC card replaced with a new device, and I installed ESXi again.
- Once the host was up again with a fresh ESXi install, I made the bare minimum configuration changes needed to make the host visible on the network; set IP information, vlan info and password.
- I reloaded the saved configuration to the host via PowerCLI...
For this step, I used:
Set-VMHostFirmware -VMHost 10.10.8.22 -Restore -SourcePath c:\Users\ewwhite\configBundle-10.10.8.22.tgz -HostUser root -HostPassword YoMama!!
Restoring the configuration forces a host reboot. Once up again, I was able to issue a reconnect to rejoin the host in vCenter and exit maintenance mode.
If PowerCLI not available, the ESXi shell commands look like:
This produces a web link that you'll be able to browse to and download a tarball of the host's configuration.
You can SCP a configuration file to a host and use the following to restore the settings.
vim-cmd hostsvc/firmware/restore_config /tmp/configBundle.tgz