We have quite a reasonably sized vSphere 'estate', with 80% of our Windows/Linux servers virtualised, running across six Datacentres. One of my challenges is medium to long term capacity planning, ensuring I get enough money into the annual Capex forecast to secure funds for either host upgrades (typically memory), more hosts (hardware and ESX licences) or worst case SAN expansion.

Anyway, up until very recently, I was quite comfortable accepting vCenter's performance statistics as being truly representative of what's going on. I typically work at the cluster level when looking at stats, as the hosts in each cluster are identically spec'd, upgraded, Etc.

However, I've recently noticed something that's bu&&ered me up a bit. One of my clusters has 200GHz of CPU "bandwidth" available, this is made up as follows:

5 hosts x 2 sockets-per-host x 6 cores-per-socket x 3.33GHz per-core = 199.8GHz

This is fine, and vCenter reports this value correctly. However, when you either view the Cluster's CPU utilization in vCenter, or pull stats using the PowerCLI's Get-Stat cmdlet, the CPU utilization can exceed 300GHz at times. This has a knock-on effect of messing up my calculations, as the utilisation figure reaches 150% (!). Now, it's a long time since I did A-level maths, but I can't see how a CPU can be 150% utilised...

So, I logged a call with VMware support. And, laughably, they said I need to buy vCenter Operations Manager (vCOPS) to do what I'm trying to do. Well, no thanks, if I have some accurate statistics, I can do my own decision support (sorry, rant over).

So, I pushed for an explanation, and the support guy said that the data in vCenter is based on a "generic" calculation that uses the sum of averages. Well, averaging data samples is quite normal and quite acceptable, but I still can't understand how you can exceed 100%.

So, I've been trying to work this out myself, and I'm wondering whether the Xeon's hyper-threading or "turbo" feature is affecting the results. However, the "turbo" up-lift is only from 3.33GHz to 3.6GHz, i.e.: 8%.

Any clues?

  • I'd guess Hyperthreading (I've never seen this firsthand, though, because I've never gotten a cluster's CPU use that high). What's the CPU resource total for the cluster show in its summary tab? – Shane Madden Oct 13 '13 at 18:32
  • Hi Shane. That's where vCenter shows 199GHz. I should probably point out that we don't see sustained utilisation like this, but we see regular periods of spikes this high (typically during backups, A/V scans or Oracle batch processing). – Simon Catlin Oct 13 '13 at 18:54
  • You should drop into chat and talk with ewwhite and chopper3. – user9517 Oct 13 '13 at 21:30
  • VMware support chap today said he does "break-fix", so won't be able to assist further. I've asked whether anyone from the engineering side of things can answer. Also, carried out some more digging, and yes, this cluster has hyper-threading enabled on all hosts, and the total MHz shown for each individual host does exceed the non-HT MHz (it's actually x2). – Simon Catlin Oct 14 '13 at 16:38
  • 1
    At this point, I hang my head in shame. Just triple-checked, and my total available cluster MHz figure in vCenter DOES include the hyper-threading. However, the value I'm using in my calculations (pulled using Power-CLI) DOESN'T cater for HT (it uses the NumCpuPackages and NumCpuCores attributes of the <hostview>.Hardware.CpuInfo object). This is consistent with the per-host view in vCenter. Thank you for input folks. I'll get my coat. I can change my extract code to pick up the NumCpuThreads attribute. – Simon Catlin Oct 14 '13 at 17:04

This is where vCenter Operations Manager can be handy. Don't discount its usefulness... it can potentially be a better DSS platform than you :) However, as with most VMware environments, you'll exhaust your RAM resources long because you run into CPU limitations. In my planning efforts with other large clusters, I would size for RAM and storage needs, as the CPU was never a limiting factor. What versions of ESXi, vSphere and license tier are in use here?

For your hosts, they sound like 3.33GHz Westmere X5680-based systems. You have an option of running those with Hyperthreading on or off. It sounds like there's something else at play here. What do the other server vitals looks like at the times the CPU spikes to 150%?

enter image description here

There is a free tier of vCenter Operations available. There's also a full-featured (60 or 90-day) evaluation available. This will be incredibly helpful in pinpointing real bottlenecks in your infrastructure... even if used to right-size VMs and validate cluster health.

The view that may make a difference for you is the "Time Remaining" metric, which calculates the amount of time remaining until a specific resource is exhausted.

enter image description here

  • Thanks for that - appreciated. The cluster in question is quite healthy most of the time, but it has a flat-our period that coincides with daily backups, a/v scans and a chunk of "offline" (i.e.: non-business hours) Oracle work (a lot of financial reconciliation). Aaaannyway, the same cluster also has a lull each day, and I'm investigating what workload can be moved to the lull. It's during this bit of work that I realised that the cluster was showing as >100% utilised, from a CPU perspective. I'll gather processor info tomorrow... – Simon Catlin Oct 13 '13 at 21:30
  • Yeah, I'm digging through all of of my environments and seeing if I can find an instance of a >100% value. Can you post the Performance graph that correlates? – ewwhite Oct 13 '13 at 21:35

To use a technical term, I co*ked up here. It turns out that the vCenter figures do indeed include Hyper-threading when it comes to total MHz. However, my spread-sheet (created using PowerCLI) wasn't picking up the "number of CPU threads", and was therefore only looking at sockets ("packages" in VMware parlance) and cores. Thank you for contributions above.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.