I have a home server with a Q6600 Quad Core and 8GB of RAM running with VMWare ESXi 3.5 for about 8 months now. I have 2 datastores, with 1TB (SATA HDs) each, one with 150GB free and the other with 240GB free. I have 9 VMs running 24x7 on it. Everything was doing great, until yesterday.
Out of the blues, I stopped getting responses from the VMs in the ESXi. At first, I could connect using the Infrastructure Client but, when if I tried to get information from any VM, I would get a message that the VM could not be reached. Looking at the Host info, it would show me networking info, cpu, memory, but when I tried to reach the datastore, it would normally stop responding. I was only able to open the datastore located where ESXi is installed once, and all the VMs were there. Now, I can't connect to it anymore, and I really don't know what to do.
Update I've done several reboots on the host, and the problem repeats itself. I connect through the Infrastructure client but, after a few seconds, it becomes unresponsive. After a while, now I can't connect through the client anymore End Update
What is the best course of action to diagnose the problem? I can access the ESXi screen without a problem, but I don't know what to do. I was thinking of reinstalling it, maybe with version 4.0, but I'm not sure I should do that. Where (and how) can I access anything that could help me figure out what's wrong?

New Update I reset the setting back to default, and I was able to connect with the VI Client. I reattached one of my VMs, and started booting it up, but I had problems again; the VM tried to boot and ended up locking up, and the VI client became unresponsive, and I couldn't connect to it again. Following @pehrs advice, I went into unsupported mode, and checked the /var/log/message, and I found a bunch of Errors reading. Below is a sample:

Aug 31 02:59:36 vmkernel: 0:00:28:41.882 cpu0:2179)StorageMonitor: 196: vmhba33:0:0:0 status =2/0 0xb 0x0 0x0
Aug 31 02:59:37 vmkernel: 0:00:28:42:357 cpu0:5279)<3>ata4: transageld ATA stat/err 0x71/04 to SCSI SK/ASC/ASCQ 0xb/00/00
Aug 31 02:59:37 vmkernel: <4>ata4: status=0x71 { DriveReady DeviceFault SeekComplete Error 0:00:28:42.357 cpu0: 5279)}
last message repeated 1 times

I also got some DriveStatusError on some lines of the same file. Now, looking at the /var/log/vmware/hostd-0.log, I'm getting some errors after successfully opening the vmdk files of the first VM that I reattached :

[2010-08-31 02:44:15:199 'PropertyCollector' 213004 warning] GetPropertyProvider failed for haTask-ha-folder-vm-vim.Folder.registerVm-45
[2010-08-31 02:45:05:693 'PropertyCollector' 98311 warning] GetPropertyProvider failed for haTask-16-vim.VirtualMachine.powerOn-49

I get several other GetPropertyProvider errors after that, then some timeouts... It seems clear I have a HD problem. What can I do to save my VMs? Can I do a scancheck on the HDs? If yes, how? Thanks! End of Update

  • What does /var/log/vmware say? – gWaldo Aug 30 '10 at 14:57
  • I'll look it up tonight, and post the response (I'm at work, and it's my home server). I can access the logs through the hidden console, right? Tks so much! – Pascal Aug 30 '10 at 16:15
  • Sounds like a hardware issue. I would strongly recommend booting up, going into unsupported mode and check the system logs. We have had similar issues with failing harddisks and motherboards on our ESXi servers. The vm's are just files and easy to migrate off the disk if it's the hardware. – pehrs Aug 30 '10 at 17:08
  • @pehrs, if it's a hardware, the log on /var/log/vmware should give some indication about it? Tks – Pascal Aug 30 '10 at 17:48
  • 1
    /var/log/messages is a good place to start looking. It should hold the vmkernel logs. – pehrs Aug 30 '10 at 18:35

I suspect that you're using consumer-grade drives for your storage? If so, these have onboard error recovery systems which will stall the volume while error recovery is being attempted. When this occurs, all storage serving can be delayed for a considerable period of time (10+ seconds).

In enterprise-grade drives, this 'feature' is disabled or never included, on the assumption that error recovery will be handled at the RAID-array level (RAID's implicitly assumed for enterprise deployments). For example, Western Digital refers to this feature (or removal of a feature!) as TLER - Time-Limited Error Recovery. In practical terms, it means a drive with TLER enabled won't stall out for an extended period of time to perform sector recovery/remapping/whatever.

So if you're running consumer drives, there's a fair chance you've hit an error on one of your disks, and it's repeatedly stalling out while it attempts to recover.

Solutions for this may be a bit tricky - I don't know if any third-party disk error scanners will support VMFS, and wouldn't risk pulling the disks and scanning them with ANYTHING, unless completely sure it won't trash the volume.

  • It's a home server, so it's a simple SATA 1TB drive. So, I guess what's left is reseting ESXi config again, and trying to copy each of the VMs on the drive, and try to savage what I can??? – Pascal Aug 31 '10 at 11:07

Why not just reboot the host machine? If you can't reboot it from the console then just power cycle it. It's a drastic measure but I've had to do it on more than one occasion.

  • I've done that... it boots, but the same problem happens again, and again... I can logon to it through the Infrastructure client, but after a few seconds, it stops responding. What can I do at the server's console to figure out what's wrong? – Pascal Aug 30 '10 at 12:02

Are you sure that you have installed the required .NET framework? I believe that the VI Client requires .NET 3.5.

I saw this at my last gig, where a lab was trying to run a standalone ESXi box. He could connect, but it would hang, disconnect, or otherwise break. We were at our wits' end, but somewhere along the way he ended up installing an additional .NET framework and that fixed the issue completely.

Yes, I know it sounds crazy.

  • I've using the VI Client for over 8 months now... and nothing changed on my machine. Plus, I can't connect to any of the VMs, so it's not a VI Client issue. But thanks... – Pascal Aug 30 '10 at 12:41
  • Ok, it was worth a shot... – gWaldo Aug 30 '10 at 14:31
  • Tks anyways! Always is! ;-) – Pascal Aug 30 '10 at 14:34
  • new answer below... – gWaldo Aug 30 '10 at 14:36
  • Tks... Appreciate some much the input... I'm kinna lost :-/ – Pascal Aug 30 '10 at 15:00

Do you have access long enough to control the VMs? If so, you might try disabling the VMs systematically in order to determine if a VM is somehow in play with this situation.

  • The last thing I was able to do when I was able to login through the VI Client was enter the host into Maintenance mode, so the guests are not booting anymore. But that still didn't resolve the problem. Now I can hardly connect to the Host anymore... and when I do, it lasts a couple of seconds before I get I get locked out – Pascal Aug 30 '10 at 13:16
  • Backup the VMs and you can troubleshoot the hypervisor and/or host without risking any additional damage to the VMs. – user48838 Aug 31 '10 at 7:42

On ESX 3.5, if you manually edited your .vmx files (like in a text editor) and you mis-typed something, it would break the VI console. Worse, it would happen to any and all VI consoles regardless of who made the mistake, and would give no message, warning, or error. (I bitched to our VMware rep about this, but they said that this was the expected behaviour...)

I recommend discovering every VM that was being modified by hand just before everything went sideways, by everyone on your team. And then validate them.

This is especially painful if you work with people who can't spell and type accurately...

  • That's so much for your input... but it's just a home server, and I'm the only one that uses it. Maybe it got corrupted. Do you think that a reset would solve that? Is a reset possible in ESXi 3.5? – Pascal Aug 30 '10 at 14:59

try SSH'ng to the machine in question.

at least if the SSH console doesn't hang up in the same amount of time you can determine if its the vSphere client / Server interface hanging up or if it's the server itsself going tits up.

Either way this sounds like a "dead" ESXi server to me. Try upgrading to a 4.0 I'd reckon as a last ditch effort and / or try to repair the install if at all possible but ...

sounds freacky

  • I can SSH'ng the machine using Putty? Is that possible in the free version? ( I have ESXi 3.5) As fas as I've read, if I upgrade to 4.0, I'll loose my guewt machines, won't I? Can I upgrade it without loosing them? – Pascal Aug 30 '10 at 16:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.