Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have a dedicated server (Core 2 Duo E4600, 2GB DDR2, LSI Raid 1 with 250GB SATA storage). running VMware ESXi 3i (3.5.0) and 3 VMs (1x Ubuntu 9.04, 1x Ubuntu 9.10, 1x Windows 2003 Web Edition)

This afternoon it suddenly stopped responding. VMware Infrastructure Client couldn't connect, Remote Desktop couldn't connect, SSH couldn't connect. Tried different internet connections etc. After a few minutes I decided to do a remote power cycle and that got everything up and running again.

Now I'm wondering: What is the right way to analyze or debug this kind of server crash?

The ESXi event log started with a clean sheet, so nothing there. The virutal machines (linux syslog, windows event logs) don't report anything special and the machine really has mediocre load overall.

What are the places to look? Can I enable more logging somewhere so I can investigate possible future crashes?

share|improve this question
1  
It turned out to be a failing hard-drive which put the RAID 1 into read-only mode, without the proper reporting going on. I'm still interested in gathering all information I can for possible future issues. Thanks for any input so far! –  user26348 Dec 1 '09 at 11:36

4 Answers 4

When rebooting after a crash, ESX usually creates a vmkernel-zdump file in the /root home directory. This is a compressed file that has an image of core and a chunk of the /var/log/vmkernel log file. The first thing to do is get the log file from this dump file

[root] vmkdump -l vmkernel-zdump-101409.14.18.1
created file vmkernel-log.1

and look at the last few lines to see if you can get any hints from the last log entries or the stack trace.

share|improve this answer
    
AFAIK there is no way to access the hypervisor filesystem remotely on ESXi. –  user26348 Dec 1 '09 at 11:35

The most obvious thing to say is "use your VMWare support contract" but of course ESXi is often used entirely freely without buying support so I'll assume you haven't.

So the next thing to do is to realise that almost certainly a system dump was created, you need to check that one was and learn how to read it. Now I could go into detail how to do this but someone has already created a nice GUIDE on how to do it.

Good luck.

share|improve this answer

You don't mention looking at the console, so I'm guessing that you are running this server headless (no monitor). This is bad - very bad - when it comes to VMWare.

Always have a monitor attached and a digital camera to take a picture of the kernel panic (wich probably was the cause). It will indicate where the panic occured - probably a faulty driver.

Edit: This could also be loss of network connectivity. If you'd have a monitor/keyboard you could log in via console and properly shut down VM's (if they have vmware tools installed) and safely reboot the host server.

share|improve this answer
4  
First of all, it is almost impossible to keep monitors attached to machines in a data center. The closest you can get is an IP KVM, but that's a large investment just to see kernel panics. Second, there is no console that allows access to the VMs in ESXi. –  MDMarra Nov 17 '09 at 12:19
    
Sorry, I was thinking ESX regarding the console. Still, having IP KVM or HP iLO in a large datacenter is far from impossible. –  pauska Nov 17 '09 at 12:36
    
It is far from impossible to attach a monitor. If you're in a large datacenter they're nearly a necessity unless you have something like iLO or DRAC. If you're cheaping out on this its easy to just setup a cart with a keyboard/mouse + lcd to wheel up and plug in when a major incident occurs. –  sparks Nov 17 '09 at 15:28
    
@pauska - Yeah my comment was specifically about monitors. IP KVMs and iLO/DRAC are pretty common and would work fine. –  MDMarra Nov 18 '09 at 4:17
    
@MDMarra - Impossible, hardly ... There's rack for that :) It's called a rack-mounted KVM, it has a slide out monitor and keyboard tray. –  JamesBarnett Jan 6 '13 at 18:30

Were you able to connect to the console via IP-KVM, remote management card, etc?

Check the amount of RAM which has been allocated to the console application. It defaults to a pretty low number. VMware support has recommended to my that I max the value out to 800 Megs.

share|improve this answer
    
That sounds like a waste of memory. 800MB is a crazy amount for just providing a configuration interface and some graphing. Do others know if this is really necessary? –  user26348 Dec 1 '09 at 11:38
    
I don't. What I will say is that my host console hasn't locked up since I made that change. I had a couple hundred days of uptime until I upgrade from 3.5 to 4.0. –  mrdenny Dec 2 '09 at 0:46
    
My boxes have 16 Gigs of RAM (at least), so loosing 800 Megs wasn't a big deal. If you can't afford the 800 Megs bump it up to what you can. –  mrdenny Dec 2 '09 at 0:47

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.