ESXi 5 forgetting configuration on reboot

Question

Everytime I reboot my ESXi 5 host it returns to an old configuration (connected to old datastores and listing old virtual machines).

I have found a few oddities which are probably part/root of the problem but i'm not sure how to go about fixing:

• /bootbank/boot.cfg is missing
• /bootbank points to /tmp
• when i issue df -h via ssh the only disks listed and NFS mountpoints, no local disk is listed

~ # df -h
Filesystem   Size   Used Available Use% Mounted on
NFS          1.4T 568.4G    828.7G  41% /vmfs/volumes/isos
NFS        119.2G  60.1G     59.1G  50% /vmfs/volumes/sql
NFS          2.7T 976.2G      1.8T  35% /vmfs/volumes/vms
~ # cd /bootbank/
/tmp # ls
imgdb.tgz    scratch      vmware-root

Any ideas on how to solve this?

Answer 1

Are you PXE booting the image? This is fairly common if you are using PXE or other remote boot (like booting on iscsi) and you aren't updating the base image when you make updates.

Answer 2

Have a look at this Vmware KB203131 article, copied below.

It occurs also with other HW! Test an installation on a local disk, this will work for sure.

---

UCS Blade loses configuration after reboot in VMware ESXi 5.0.x

Symptoms:

• UCS Blades booting from FCoE/iSCSI SAN revert to a previous state after reboot.
• The ESXi 5.0 Bootbank points to /tmp and/or no altbootbank exists.
• You are unable to install VMware Tools in virtual machines.
• You are unable to patch/update ESXi 5.0.

• A task fails with:

  • The host returns esxupdate error code: 15. The package manager transaction is not successful. Could not install patches on <hostname>
    
    

• In the esxupdate.log, you see messages similar to:

  • esxupdate: ERROR: InstallationError: ('Current bootbank /bootbank is not verified and most likely a serious problem was encountered during boot, it is not safe to continue altbootbank install. bootstate is 2, expected value is 0.')

  • esxupdate: HostImage: DEBUG: Live image has been updated but /altbootbank image has not. This means a reboot is not safe.

  • esxupdate: HostImage: INFO: Installer <class 'vmware.esximage.Installer.BootBankInstaller.BootBankInstaller'> was not initiated - reason: altbootbank is invalid: Error in loading boot.cfg from bootbank /bootbank: Error parsing bootbank boot.cfg file /bootbank/boot.cfg: [Errno 2] No such file or directory: '/bootbank/boot.cfg'

Cause:

The FCoE device discovery may take longer than expected, and when the restore-bootbank jumpstart plug-in starts, the bootbank partition on the FCoE LUN is still not available.

Resolution:

This is a known issue in VMware ESXi 5.0.x. A resolution for this issue is being evaluated for inclusion in a future release.

If you are experiencing this issue, open a Support Request.

To work around this issue, reinstall from local storage.

Answer 3

Whilst rebooting over IPMI i noticed Shift-R for recovery mode. After pressing the ESXi host booted without any datastores/virtual machines configured. I set them up and rebooted. Everything appears to be back to normal:

• /bootbank/boot.cfg exists
• df -h lists three vfat volumes as well as the NFS volumes

Thanks for the suggestions. I'm still curious to know how it broke- although glad to've resolved it without having to go to the physical location and re-install the hypervisor!

Lee