Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

An SD (SDHC) card installed in an HP ProLiant DL380p Gen8 server running VMware ESXi just failed :(

I encountered some ominous looking messages on the vCenter console and in the HP ProLiant ILO event log...

Lost connectivity to the device ... backing the boot filesystem. As a result, host configuration changes will not be saved to persistent storage.

enter image description here

Embedded Flash/SD-CARD: Error writing media 0, physical block 848880: Stack Exception.

enter image description here


VMware advocates the use of USB and SD (SDHC) boot devices for ESXi. It was one of the main reasons the smaller footprint ESXi was developed (versus the older ESX). I've spent much time highlighting the differences between ESXi's installable and embedded modes to coworkers and clients. However, these failures do seem to happen. In this case, this is my third instance.

Luckily, this is a vSphere cluster with SAN storage. What steps should be taken to remediate this failure?

share|improve this question

3 Answers 3

I'd say just reinstall ESXi fresh on a new card, it's not worth your time or effort to try to preserve the potentially corrupt data on the current card.

share|improve this answer

KB article: http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&popup=true&languageId=&externalID=2004784

Note: In the event that your original SD flash card or USB drive fails, these steps can be used in a disaster recovery capacity to get a fully functional ESXi host running as quickly as possible on a replacement USB drive or SD flash card.

Disaster Recovery Scenario

If you have a local datastore containing production information and have completed the installation on a replacement USB flash drive or SD flash card in a disaster recovery scenario, you may be required perform these additional steps:

  1. Connect to the ESXi host using the vSphere Client or add the ESXi host to an existing vCenter Server implementation.
  2. Click the host.
  3. Click the Configuration tab.
  4. Click Storage.
  5. Click Rescan to ensure you can see all datastores.
  6. Browse the datastore and register any virtual machines that are located on the datastores.
share|improve this answer
up vote 11 down vote accepted

Here's the process I used to resolve this:

VMware ESXi can be installed in an embedded mode or an installable mode. As outlined here, the installation mode is determined by the destination media and the size of the volume available to the ESXi installer.

  • USB, SDHC or any device less than 5GB in size: Embedded
  • Hard drives/volumes greater than or equal to 5GB in size: Installable

One of the unique attributes of running ESXi in embedded mode is that the OS is loaded into RAM and only touches the USB/SD device hourly during normal operation. In my situation, the system continued to operate, even with a failed SDHC device.

The error message I received in the vCenter interface indicated that configuration changes would not be saved, but the cluster was still usable. I left the system in this state for several days until I could get to the datacenter to replace the SD card.


With regard to steps to take following a failure of a USB or SD device, it is important to extract and save a copy of your host's settings!!

This is easily accomplished via PowerCLI or the vSphere CLI. I used PowerCLI running from the vCenter server:

Get-VMHostFirmware -VMHost 10.10.8.22 -BackupConfiguration -DestinationPath C:\Users\ewwhite\Downloads
  • Following that, I evacuated all virtual machines from the affected host and placed it in maintenance mode.
  • The host was then shut down, the SDHC card replaced with a new device, and I installed ESXi again.
  • Once the host was up again with a fresh ESXi install, I made the bare minimum configuration changes needed to make the host visible on the network; set IP information, vlan info and password.
  • I reloaded the saved configuration to the host via PowerCLI...

For this step, I used:

Set-VMHostFirmware -VMHost 10.10.8.22 -Restore -SourcePath c:\Users\ewwhite\configBundle-10.10.8.22.tgz -HostUser root -HostPassword YoMama!!

Restoring the configuration forces a host reboot. Once up again, I was able to issue a reconnect to rejoin the host in vCenter and exit maintenance mode.

That's all!

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.