2

I installed ESXi 4.0 on an HP Proliant g5 with a 64bit Xeon processor and took advantage of the free license as I work for a public school. I created two instances of server 2003 from scratch, one to be the DC, DHCP, the other to be a file server and DNS/DHCP backup. I had both guests up and running fine, setup my user accounts, transferred the data, etc etc.

Once I joined a client machine to the domain, I would find that both of my Windows guests would lock up. Sometimes it would be for five or so minutes, once it was overnight. The "locked up" state means that as far I could tell, all services were stopped; dhcp no longer handed out IP's, DNS stopped working, I couldn't RDP into the server. The ESXi host, my HP server, was still running fine. VSphere was working, and I could look at the performance of the individual guests.

I would try Powering off the hosts from inside VSPhere, and the hosts would start powering off, but get stuck at 95%, and stay that way, sometimes only for 10 minutes, others for hours. Several times I had to restart ESXi from it's console in order to restart my machines.

Now, can anyone tell me what is happening, and how I can fix it, or take steps to prevent it? I hired a consultant to come take a look at it, someone who's experience and knowledge I trust, and he told me he had never seen anything like this ever before. He spoke to a friend of his who is VM certified, and he also said he had never heard of this issue. Thanks for your replies, and I'll do my best to respond ASAP. Currently, the server is powered off, and I've reinstituted my nine year old Server 2000 boxes, and I'm considering installing ESXi 3.5. Does anyone know a host created in 4.0 will work in 3.5? I'd really like to avoid having to rebuild those accounts! I know 4.0 works on this server, as I have another server in another school with the same exact hardware running 4.0 fine.

  • Brendan
1
  • If you're forced to go down to 3.5, you can import the VMs by running them through VMWare Convertor Standalone. Apr 11, 2010 at 21:15

8 Answers 8

2

This seems like a hardware issue to me, e.g., bad ram. Try running the Offline Diagnostics disc for your server model (if you don't have one, you can download an ISO from hp.com)

1

Are the logs showing anything? What about network activity for the VM's from VSphere?

My next thing I'd try doing is to install some kind of packet sniffer if nothing is showing up in the logs. If the system is at all responsive you can try running wireshark right on the system acting up, see if it will update the screen before slowing or locking up. Maybe running tcpmon from sysinternals might give a clue.

Otherwise, try setting up a VM with Linux and direct it (or redirect network traffic from the VM's) through that to see what it can see with wireshark.

If network traffic is goign crazy you may have to find some way to sniff what's happening; if it were just a name conflict or some kind of AD replication issue it would be in the logs.

We see degradation when there's heavy backup processes going on on the network, but you didn't mention anything about replicating files or anything like that.

Is this feasible in your situation?

4
  • Whoops, sorry, forgot to mention the logs. I've noticed no errors or problems in the Windows or ESX logs. Network traffic did seem to go pretty crazy during a lockup. Sep 14, 2009 at 14:56
  • I edited my reply a bit... Sep 14, 2009 at 15:01
  • I was very careful to not overlap jobs while building this. It happened to me twice, the first time I assumed it was a bad server installition, and blew both servers away and started anew. That said, file transfers were done, backup was done the night before on the older servers, and there were no users logged onto any clients throughout the school, as I did most of this work during the summer and have only recently learned of ServerFault. I'm not sure what you mean by "Setting up a VM with Linux and direct [traffic] trhough it" I'm assuming you mena a Linux Host acting as proxy? Sep 14, 2009 at 15:07
  • Linux guest acting as a proxy. Forward traffic from the systems that are misbehaving to the Linux system that then transparently forwards it to the actual router, then try sniffing traffic. Sep 14, 2009 at 15:30
1

When this happens look can you get onto a console on the host (not sure if ESXi gives you a console or not) and look to see if the process is orphaned or not. If the process which is the VM is orphaned then you'll have to reboot the host to clear the process.

I've seen this happen a couple of times on ESX 3.5 and 4.0. If the guests are upgrades from 3.5 then you need to make sure that the hardware version has been upgraded, as well as the client tools. I assume that you did install the client tools on the guests?

1
  • The guests were created on 4.0, and the first thing I do is install the client tools, otherwise the cursor is nearly useless. I'll need to see if the process is orphaned next time it happens. Sep 15, 2009 at 16:53
1

I've no reason to suspect that there is a compatibility problem but have you checked the servers themselves and all of the component hardware (especially NICs, seen lots of problems with NICs in my time) for compatibility with ESXi 4 VMware HCL?

1
  • I just checked the HCL, and the server as well as the NIC is fully compatible with ESXi 4, as well as 3.5. Sep 15, 2009 at 16:43
1

Is this a dual-core CPU? How did you configure your virtual machines, how man vCPU did you choose for each? I know for a fact that 3.5 has issues with their bootup-time if you selected more than 1 vCPU on each machine, and you didn't really gain any performance from it anyway.

0

I've encountered situations in the past where something in a group policy resulted in a machine periodically locking up (in my case it was applying Vista specific GPOs for power management to a Win 7 machine IIRC), so I'd suggest a quick look to see if joining the domain has caused a subtle problem.

0

Check the following:

  • Any snapshots currently left open on the guest VM? Snapshots on DCs are not a good idea, but in general leaving a snapshot open for a long time can lead to VM lockups, especially on DCs SQL and Exchange servers.

  • Any unusual hardware attached to the VM e.g. Floppy disk, USB passthrough or serial ports? Strip the devices down to the absolute minimum you require.

  • Run a suite of tests on the server's hardware. There's a decent suite on HP's smartstart for the G5s. If you still have support on the hardware, call HP and see if their support folks have any advice (they're pretty good, IMO).

  • Swap out the RAM sticks with another set, if you have some available.

What kind of disks are the VMs being run from? SAN or local? Stock controller, or a discrete one? Have you ruled out problems with the install media?

Edit: just remembered... check your NIC settings on the host server. I vaguely remember having problems with one of the NIC features (TCP offloading?) being enabled on the host NIC, and needing to disable it in ESX 4.0.

0

I have the related issue on the above (ESXi 4) but it only happened to the 3rd VMs which hang and can't shutdown but stuck at 95%. Noticed the problem was due to SEP10 but the 1st and 2nd vm which installed with Symantec doesn't have the problem except the 3rd. Delete vm reinstall and it still the same and always be the 3rd vm.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .