Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

How do I find out which VM is responsible for the majority of the I/O I see with Hyper-V?

The situation: I do have Hyper-V hosts with a number of VMs (around 30 per host) where I occasionally see prolonged periods of high disk utilization. I would like to know which VM is causing this.

I tried using Process Explorer, but it reported all Hyper-V related I/O to be handled by the "System" process, so no help there:

process explorer

It already would help to have a breakdown by files accessed. The full GUI install of Windows Server has the Resource Monitor which is providing this information. However, Resmon seems to be unavailable in Core and Hyper-V installations.

share|improve this question
    
@RyanRies no, I do not see vmwp.exe involved in any of the I/O. As I do not know which disks the load is going to, I can't tell if it is taking the IDE or the SCSI code path, both seems equally possible. –  the-wabbit May 28 '13 at 13:34

3 Answers 3

up vote 6 down vote accepted

Open Performance Monitor (run as Admin) on your local workstation. Add counter, select counters from the hyper-v machine, Hyper-V Virtual IDE Controller or Hyper-V Virtual Storage Device, select counters and instances (VMs) as seems appropriate. You might have to fish around a bit to find the counter that makes the most sense to you.

There are some good hints as to which Counters to look for in Monitoring Hyper-V Performance

share|improve this answer
    
Mmh, I tried that. It did not yield useful results and I have no idea why this is so. At first, I had trouble locating the Virtual Storage Device counters - it seems like they are not present on my installs. Since most of my disks are IDE, I gave the Virtual IDE Controller a shot but for some unfathomable reasons it is not returning anything but zeros for the instance actually causing the load (which I stumbled upon by accident in the meantime) - whereas other instances seem to produce sane values. –  the-wabbit May 31 '13 at 20:37
2  
If you have the VM Integration stuff installed then you're not using IDE (regardless of what the configuration screen says; don't get me started on MS's poor config screen choices), so you have to use the Virt Stor Dev counters. The latter are per VHD, not per VM, so there's a little bit of dereferencing involved. –  Chris S May 31 '13 at 22:32
    
This clears it up. I finally found the virtual storage device, it seems to return sane values for the virtual disks. –  the-wabbit Jun 3 '13 at 9:31
    
Chris S is right. The exact name of the performance monitor counter he refers to is "Hyper-V Virtual Storage Device". I.e. the "Queue length" in this category gives a nice insight into which of your VHD(X) files is having most latency communicating with your disk. –  Koen Zomers Jan 8 at 13:55

Syneticon-dj, I wrote something for you this afternoon. I thought this problem was interesting, so this simple script will give you the read and write IO stats on each running VM on the Hyper-V host. As an added bonus it associates each VM to its vmwp.exe's Process ID.

You can run this on your Hyper-V server, because it doesn't need a GUI.

The downside is that while I was working on this, I noticed that the performance counters were working great for a while, and then for no discernible reason they decided to all stay on zero. Well maybe it's not a bug, like Chris S says... but these counters might unfortunately not be very useful after all. Regardless, it would be very easy to modify the script to use the Virt. Storage Device counters instead.

The output looks like this:

PID     VMName               ReadBytesPerSec             WriteBytesPerSec
---     ------               ---------------             ----------------
5108    DC02                          483.90                            0
2796    DC01                               0                            0
3348    ECA01                     4782668.27                            0

#Requires -Version 3
function Get-VMPidAndIO
{
<#
.SYNOPSIS
    Gets the Process ID and I/O statistics of each virtual machine running on the Hyper-V host.
.DESCRIPTION
    Gets the Process ID and I/O statistics of each virtual machine running on the Hyper-V host.
    Currently only works for VMs using virtual IDE controllers.
    Requires Powershell 3 at a minimum.
.LINK
    http://myotherpcisacloud.com
.NOTES
    Written by Ryan Ries, June 2013.
    ryan@myotherpcisacloud.com
#>
    BEGIN
    {
        Try
        {
            $VMProcesses = Get-CimInstance -Query "Select ProcessId,CommandLine From Win32_Process Where Name ='vmwp.exe'" -ErrorAction Stop
        }
        Catch
        {
            Write-Error $_.Exception.Message
            Return
        }
    }
    PROCESS
    {

    }
    END
    {
        Foreach($_ In $VMProcesses) 
        {
            $VMName = $((Get-VM | Where Id -EQ $_.CommandLine.Split(' ')[-1]).Name)            
            [PSCustomObject]@{PID=$_.ProcessId;
                              VMName=$VMName; 
                              ReadBytesPerSec=[Math]::Round($(Get-Counter "\Hyper-V Virtual IDE Controller (Emulated)($VMName`:Ide Controller)\Read Bytes/sec").CounterSamples.CookedValue, 2);
                              WriteBytesPerSec=[Math]::Round($(Get-Counter "\Hyper-V Virtual IDE Controller (Emulated)($VMName`:Ide Controller)\Write Bytes/sec").CounterSamples.CookedValue, 2); }
        }

    }
}
share|improve this answer
    
That's not a bug, see comments in my answer. –  Chris S Jun 3 '13 at 13:03
    
Then I wonder why the counters seem to intermittently work? (Windows guests with full integration services.) And why they would choose the significantly less useful level of abstraction of per-VHD counters instead? :( –  Ryan Ries Jun 3 '13 at 13:10
    
Not sure why it would show anything intermittently. As for why, it's the way storage works in Hyper-V. Essentially there is only one storage controller for all VMs. By using an "IDE Controller" in the configuration you are just enabling IDE Emulation for the storage exposed to that VM (as opposed to the "SCSI Controller" that has nothing to do with SCSI, and just disables any sort of storage controller emulation). VMs that support Integration Services just skip the IDE emulation. The storage controller counters don't exactly care what VM is attached to each VHD, hence the way it works. –  Chris S Jun 3 '13 at 13:19
    
Awesome. Thanks for the info. I'm extremely interested in reading more about this, but I'm having a hard time turning up any good technical documentation. I think it would behoove MS to demystify a lot of this stuff for us. We'd be able to put more confidence in their solutions. I just want a Hyper-V Internals book. Is that too much to ask? :) –  Ryan Ries Jun 3 '13 at 13:37

Both of the other answers are useful. But I find that I get what you're looking for much more easily (when the VHDs are stored on local or locally shared disks instead of a file server) by opening up "Resource Monitor" and looking at the Disk tab. Look at "Disk Activity" and sort by "Total (B/sec)." You'll see the VHDs listed in order of their activity.

Obviously my strategy is a loose approximation for machines that you're locally logged into, which makes it less useful in a big, headless environment.

share|improve this answer
1  
Jake, thank you for taking the time to respond. I'd really love to just look at the resource monitor, however, it is unavailable when the server is not running a full graphical install but only the Hyper-V edition or Server Core. Which is what I have. –  the-wabbit May 31 '13 at 18:02

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.