What happens when the USB key or SD card I've installed VMware ESXi on fails?

Question

An SD (SDHC) card installed in an HP ProLiant DL380p Gen8 server running VMware ESXi just failed :(

I encountered some ominous looking messages on the vCenter console and in the HP ProLiant ILO event log...

Lost connectivity to the device ... backing the boot filesystem. As a result, host configuration changes will not be saved to persistent storage.

Embedded Flash/SD-CARD: Error writing media 0, physical block 848880: Stack Exception.

---

VMware advocates the use of USB and SD (SDHC) boot devices for ESXi. It was one of the main reasons the smaller footprint ESXi was developed (versus the older ESX). I've spent much time highlighting the differences between ESXi's installable and embedded modes to coworkers and clients. However, these failures do seem to happen. In this case, this is my third instance.

Luckily, this is a vSphere cluster with SAN storage. What steps should be taken to remediate this failure?

Answer 1

Here's the process I used to resolve this:

VMware ESXi can be installed in an embedded mode or an installable mode. As outlined here, the installation mode is determined by the destination media and the size of the volume available to the ESXi installer.

• USB, SDHC or any device less than 5GB in size: Embedded
• Hard drives/volumes greater than or equal to 5GB in size: Installable

One of the unique attributes of running ESXi in embedded mode is that the OS is loaded into RAM and only touches the USB/SD device hourly during normal operation. In my situation, the system continued to operate, even with a failed SDHC device.

The error message I received in the vCenter interface indicated that configuration changes would not be saved, but the cluster was still usable. I left the system in this state for several days until I could get to the datacenter to replace the SD card.

---

With regard to steps to take following a failure of a USB or SD device, it is important to extract and save a copy of your host's settings!!

This is easily accomplished via PowerCLI or the vSphere CLI. I used PowerCLI running from the vCenter server:

Get-VMHostFirmware -VMHost 10.10.8.22 -BackupConfiguration -DestinationPath C:\Users\ewwhite\Downloads

• Following that, I evacuated all virtual machines from the affected host and placed it in maintenance mode.
• The host was then shut down, the SDHC card replaced with a new device, and I installed ESXi again.
• Once the host was up again with a fresh ESXi install, I made the bare minimum configuration changes needed to make the host visible on the network; set IP information, vlan info and password.
• I reloaded the saved configuration to the host via PowerCLI...

For this step, I used:

Set-VMHostFirmware -VMHost 10.10.8.22 -Restore -SourcePath c:\Users\ewwhite\configBundle-10.10.8.22.tgz -HostUser root -HostPassword YoMama!!

Restoring the configuration forces a host reboot. Once up again, I was able to issue a reconnect to rejoin the host in vCenter and exit maintenance mode.

---

If PowerCLI not available, the ESXi shell commands look like:

vim-cmd hostsvc/firmware/backup_config 

This produces a web link that you'll be able to browse to and download a tarball of the host's configuration.

You can SCP a configuration file to a host and use the following to restore the settings.

vim-cmd hostsvc/firmware/restore_config /tmp/configBundle.tgz

That's all!

Answer 2

KB article: http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&popup=true&languageId=&externalID=2004784

Note: In the event that your original SD flash card or USB drive fails, these steps can be used in a disaster recovery capacity to get a fully functional ESXi host running as quickly as possible on a replacement USB drive or SD flash card.

Disaster Recovery Scenario

If you have a local datastore containing production information and have completed the installation on a replacement USB flash drive or SD flash card in a disaster recovery scenario, you may be required perform these additional steps:

1. Connect to the ESXi host using the vSphere Client or add the ESXi host to an existing vCenter Server implementation.
2. Click the host.
3. Click the Configuration tab.
4. Click Storage.
5. Click Rescan to ensure you can see all datastores.
6. Browse the datastore and register any virtual machines that are located on the datastores.

Answer 3

I'd say just reinstall ESXi fresh on a new card, it's not worth your time or effort to try to preserve the potentially corrupt data on the current card.

Answer 4

One of my hosts showed the same "Lost connectivity" error. I googled and found this post. So I tried to backup the configuration with PowerCLI and vCLI as suggested. Backup came back with "Internal error". Called VMWare. The answer is that when the SD failed, you won't be able to backup your configuration because the backup command can only pull the configuration from the SD card.