1

I am having a serious problem with a server running ESXi 6.0 Everything worked great last week. Now out of no where the whole thing is basically useless. I am getting datastore latency of up to 51 seconds! Nothing has changed between now and last week other than installing some software on a VM.

Datastore Lag

The server is an HP Proliant DL360 G7 2X hexacore Xeon X5650 2.67GHz 144 GB RAM. 8x 300GB HP 10k SAS hard drives in RAID 10.

I have 6 VM's on the machine, most with thin provisioned VMDK's Out of 1.6 TB I have 600GB free.

2 of the VM's seem to run fine, the others run like total crap.

I have tried rebooting the server. Assigning more resources to the slow VM's (even tho they have plenty) and nothing is working.

Even with every VM powered off I have tried to move the VM's off the server to a storage device on the network and I am getting spikes in data transfer. It will move at 20 -30MB /s for about 20 seconds then drop down to near 0 for a few minutes then back up in a constant pattern which suggests a bottleneck somewhere.

When I try to move data between virtual drives in a powered on VM, same thing happens. Right now I am trying to transfer a file and it is going about 200kb/s. On the slow VM's it takes over 20 minutes to boot and is so slow you can't use it.

Disk transfer rate

I am at a total loss. Any help in resolving this would me much appreciated.

2
  • Is this a standalone host or part of a cluster managed by vCenter? Can you post the version and build number of your ESXi host? Was this installed with the HP-specific version of ESXi?
    – ewwhite
    Aug 29, 2016 at 0:37
  • IT is a standalone host. It was not installed with the HP version of ESXi. It is ESXi 6.0.0 3620759 Aug 29, 2016 at 0:41

1 Answer 1

3

I would suggest that your issue is related to the health of your RAID controller's cache and battery/flash module. If the RAID write cache has been disabled due to failure of the RAID battery, for instance, your write performance on the array will degrade severely.

There are a couple of ways to check this. Can you specify if this is a standalone host or part of a cluster managed by vCenter?


Edit:

This host does not appear to have the HP-specific version of ESXi installed.

Without this or the HP add-ons for ESXi, there's no monitoring of the host hardware or any of the utilities necessary to check system status.

Normally, you can see status graphically like this:

enter image description here

enter image description here

I suspect you have a storage battery failure, considering the G7 line was introduced in 2011 and the batteries tend to last 3-5 years in production. If this was a used server, this is the likely cause. You should add them from here, here and here.

At the command line, running the following will show your battery's status (other handy commands):

/opt/hp/hpssacli/bin/hpssacli ctrl all show config detail | grep -i battery

Output:

[root@c2-esx1:~] /opt/hp/hpssacli/bin/hpssacli ctrl all show config detail | grep -i battery
   No-Battery Write Cache: Disabled
   Battery/Capacitor Count: 1
   Battery/Capacitor Status: OK

If the part is bad, we can force it to ignore the battery status using the following (there's risk if you don't have stable power for your equipment):

/opt/hp/hpssacli/bin/hpssacli ctrl slot=0 modify nbwc=enable

This will at least restore performance while you arrange for parts repair/replacement.

21
  • This is a standalone host. It is not managed by vcenter server. Its just ESXi on a server and I connect to it with the Vsphere client. (not the web client) Aug 29, 2016 at 0:39
  • Please add the build number and let us know if you installed the HP-specific version of ESXi.
    – ewwhite
    Aug 29, 2016 at 0:40
  • It is not the HP specific version. It is ESXi 6.0.0 3620759 Aug 29, 2016 at 0:42
  • @KyleVaughan The HP version definitely needs to be installed for actual monitoring and to have tools to work on the server. But you probably have a RAID battery failure.
    – ewwhite
    Aug 29, 2016 at 1:15
  • I tried to install the HP version but it wouldn't boot so I went with the standard version. I am currently in the process of installing the HP array controller drivers and the utilities ( rebooting now ) will report back with my findings. Hopefully the server boots up as I have no physical access to it. (it's on the other side of the country ) Aug 29, 2016 at 1:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .