4

We have recently started moving our C++ build server(s) from real machines into VMs. (MS Hyper-V)

We have some performance issues that I've currently no idea how to address. We have:

  • Test-Box - this is a piece of desktop workstation hardware my co-worker used to set up the VM before we moved it to the actual server hardware
  • Srv-Box - this is the server hardware
  • Test-Box-Real - This is Windows running directly on the Test-Box HW
  • Test-Box-VM - This is Windows in a Hyper-V VM on the Test-Box HW
  • Srv-Box-Real- This is Server2008R2 running on the Srv-Box HW.
  • Srv-Box-VM- This is Windows running in a Hyper-V VM on the Srv-Box HW, i.e. on Srv-Box-Real.

Now, the problem is that we compared Build times between Test-Box-Real and Test-Box-VM and they were basically equal (within about 2%).

Then we moved the VM to the Srv-Box machine and what we saw there is that we have a significant performance degradation between Srv-Box-Real and Srv-Box-VM, that is, where we saw no differences on the Test HW we now do see major differences in performance on the actual Server HW. (Builds about ~~ 50% slower inside the VM.)

I should add that both the Test-Box and the Srv-Box are only running this one single VM and doing nothing else.

I should also note that the "Real" OS is Win2008R2(64bit) and the VM hosted OS is Wind2003R2(32bit).

Hardware specs:

  • Srv-Box:

    • Intel XEON E5640 @ 2.67Ghz (This means 8 cores with hyperthreading on the Real system and "only" 4 cores on the VM, since Hyper-V doesn't allow for hyperthreading, but number of cores doesn't seem to explain the problem here.)
    • 16GB RAM (we have 4GB assigned to the VM)
    • Virtual DELL RAID 1 (2x 450GB HUS156045VLS600 Hitachi 15k SAS drives)
  • Test-Box:

    • Intel XEON E31245 @ 3.3GHz
    • 16GB RAM
    • WD VelociRaptor 600GB 10k RPM SATA

Note again that I'm only concerned with the differences between Srv-Box-Real and Srv-Box-VM (high) vs. the differences seen btw. Test-Box-Real and Test-Box-VM (low).

Why would one machine have parity when comparing VM vs Real performance and the other (server grade HW no less) would have a large disparity? (Both being XEON CPUs ...)

3
  • How much RAM did you assign to Test-Box-VM? Was this also 4GB? If your build process needs that much RAM, less RAM (4 vs 16GB is a big difference) could cause swapping which can have quite a severe performance impact.
    – brain99
    Sep 3, 2012 at 11:01
  • I can not explain 50% with this, but check if the server has VT-x and VT-d enabled. It might at least explain part of that huge difference. Also check IO speeds inside and outside the VM.
    – Hennes
    Sep 3, 2012 at 11:35
  • Quite likely IO based - the 15k SAS discs are most likely the bottleneck.... unless there is no other VM doing IO.
    – TomTom
    Sep 3, 2012 at 12:19

3 Answers 3

2

Start with the obvious - monitor performance counters, if possible using Ressource MOnitor. How is CPU, IO handling? The standard" bottleneck is pretty much IO, and your 2 drives sound like quite bad IO for anything that has more then 2-3 VM's, especially when something brutal as a build server for C++ hits it (that is HEAVY IO based, compilers in generally are ,but C++ is quite extensive on this area - I use a SSD just for the build).

This is standard Admin - nothing special about a VM at this point. Basically to find the bottleneck you first find it.

IO may by also - dynamic discs. Helps to put in a non dynamic disc ONLY for the build step (work files). There is a TON of IO happening there.

Now, yes, a Raptor has less IOPS than a 15k SAS, but then it is likely NON SHARED.

Does SRV-HW use up all CPU cores? Another item, then...

Regarding your CPU limit, btw., the CURRENT version of Hyper-V supports a LOT more cores than 4 per VM, so an upgrade may be in order... Server 2012 is available for pretty much everyone not buying a box in a shop right now, I got my copy 16th of August from the MS repositories.

But first you do normal operational step by step - find out where the bottleneck is by using ressource monitor and performance

6
  • Only one single VM in all setups. That's exactly the reason why we can't make anything of this. But of course you're right, we should start by monitoring, but that a ton more work than trying to get some ideas first here abouts :-)
    – Martin
    Sep 3, 2012 at 12:55
  • The idea normally is: Check. takes only some minutes to check hosst side: CPU, IO, Memory loads. Normalyl it is, in this order: IO, CPU, Memory, that overload.
    – TomTom
    Sep 3, 2012 at 15:35
  • Yeah, but if you look at the stats I posted, it seems totally weird that for the SAS-15kRPM RAID System the performance inside the (single) VM is significantly different from "outside" while for the SATA-10k non-RAID System the VM performance and the "real" performance are on par. ... ... Then again, performance issues are always weird it seems :-)
    – Martin
    Sep 3, 2012 at 19:01
  • Yes. But with a little 4 minute test you KNOW it is the IO or not ;)
    – TomTom
    Sep 4, 2012 at 4:51
  • TomTom ... 4 minutes? I doubt I can pull this off in 4 minutes :-) ... Maybe you can help some: serverfault.com/questions/424126/…
    – Martin
    Sep 4, 2012 at 11:03
1

I had a similar problem in the past, we tracked this down to write caching Disable it on your physical server and give it another go.

The other thing to keep in mind is your simply battling a lot of differences,

  1. VM
  2. Different memory
  3. Different disks (VHD vs Physical)

Eliminate as many as you can and try again.

0

Check the BIOS settings wrt. performance. (We changed it from "Custom" to "Full Performance".)

We found out that on this server, the system did not detect full load when the full load came from within the Hyper-V VM, meaning that the processor bumbled along at 1,6GHz instead of running at it's full potential. Speed Stepping did work when running on the "real" OS -- that's where the difference came from as far as we can tell.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .