3

I have a dedicated server (Core 2 Duo E4600, 2GB DDR2, LSI Raid 1 with 250GB SATA storage). running VMware ESXi 3i (3.5.0) and 3 VMs (1x Ubuntu 9.04, 1x Ubuntu 9.10, 1x Windows 2003 Web Edition)

This afternoon it suddenly stopped responding. VMware Infrastructure Client couldn't connect, Remote Desktop couldn't connect, SSH couldn't connect. Tried different internet connections etc. After a few minutes I decided to do a remote power cycle and that got everything up and running again.

Now I'm wondering: What is the right way to analyze or debug this kind of server crash?

The ESXi event log started with a clean sheet, so nothing there. The virutal machines (linux syslog, windows event logs) don't report anything special and the machine really has mediocre load overall.

What are the places to look? Can I enable more logging somewhere so I can investigate possible future crashes?

|improve this question
  • 1
    It turned out to be a failing hard-drive which put the RAID 1 into read-only mode, without the proper reporting going on. I'm still interested in gathering all information I can for possible future issues. Thanks for any input so far! – user26348 Dec 1 '09 at 11:36
6

When rebooting after a crash, ESX usually creates a vmkernel-zdump file in the /root home directory. This is a compressed file that has an image of core and a chunk of the /var/log/vmkernel log file. The first thing to do is get the log file from this dump file

[root] vmkdump -l vmkernel-zdump-101409.14.18.1
created file vmkernel-log.1

and look at the last few lines to see if you can get any hints from the last log entries or the stack trace.

|improve this answer
  • AFAIK there is no way to access the hypervisor filesystem remotely on ESXi. – user26348 Dec 1 '09 at 11:35
2

The most obvious thing to say is "use your VMWare support contract" but of course ESXi is often used entirely freely without buying support so I'll assume you haven't.

So the next thing to do is to realise that almost certainly a system dump was created, you need to check that one was and learn how to read it. Now I could go into detail how to do this but someone has already created a nice GUIDE on how to do it.

Good luck.

|improve this answer
0

Were you able to connect to the console via IP-KVM, remote management card, etc?

Check the amount of RAM which has been allocated to the console application. It defaults to a pretty low number. VMware support has recommended to my that I max the value out to 800 Megs.

|improve this answer
  • That sounds like a waste of memory. 800MB is a crazy amount for just providing a configuration interface and some graphing. Do others know if this is really necessary? – user26348 Dec 1 '09 at 11:38
  • I don't. What I will say is that my host console hasn't locked up since I made that change. I had a couple hundred days of uptime until I upgrade from 3.5 to 4.0. – mrdenny Dec 2 '09 at 0:46
  • My boxes have 16 Gigs of RAM (at least), so loosing 800 Megs wasn't a big deal. If you can't afford the 800 Megs bump it up to what you can. – mrdenny Dec 2 '09 at 0:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy