13

How do I find out which VM is responsible for the majority of the I/O I see with Hyper-V?

The situation: I do have Hyper-V hosts with a number of VMs (around 30 per host) where I occasionally see prolonged periods of high disk utilization. I would like to know which VM is causing this.

I tried using Process Explorer, but it reported all Hyper-V related I/O to be handled by the "System" process, so no help there:

process explorer

It already would help to have a breakdown by files accessed. The full GUI install of Windows Server has the Resource Monitor which is providing this information. However, Resmon seems to be unavailable in Core and Hyper-V installations.

1
  • @RyanRies no, I do not see vmwp.exe involved in any of the I/O. As I do not know which disks the load is going to, I can't tell if it is taking the IDE or the SCSI code path, both seems equally possible.
    – the-wabbit
    May 28, 2013 at 13:34

5 Answers 5

9

Open Performance Monitor (run as Admin) on your local workstation. Add counter, select counters from the hyper-v machine, Hyper-V Virtual IDE Controller or Hyper-V Virtual Storage Device, select counters and instances (VMs) as seems appropriate. You might have to fish around a bit to find the counter that makes the most sense to you.

There are some good hints as to which Counters to look for in Monitoring Hyper-V Performance

4
  • Mmh, I tried that. It did not yield useful results and I have no idea why this is so. At first, I had trouble locating the Virtual Storage Device counters - it seems like they are not present on my installs. Since most of my disks are IDE, I gave the Virtual IDE Controller a shot but for some unfathomable reasons it is not returning anything but zeros for the instance actually causing the load (which I stumbled upon by accident in the meantime) - whereas other instances seem to produce sane values.
    – the-wabbit
    May 31, 2013 at 20:37
  • 2
    If you have the VM Integration stuff installed then you're not using IDE (regardless of what the configuration screen says; don't get me started on MS's poor config screen choices), so you have to use the Virt Stor Dev counters. The latter are per VHD, not per VM, so there's a little bit of dereferencing involved.
    – Chris S
    May 31, 2013 at 22:32
  • This clears it up. I finally found the virtual storage device, it seems to return sane values for the virtual disks.
    – the-wabbit
    Jun 3, 2013 at 9:31
  • Chris S is right. The exact name of the performance monitor counter he refers to is "Hyper-V Virtual Storage Device". I.e. the "Queue length" in this category gives a nice insight into which of your VHD(X) files is having most latency communicating with your disk. Jan 8, 2014 at 13:55
2

Syneticon-dj, I wrote something for you this afternoon. I thought this problem was interesting, so this simple script will give you the read and write IO stats on each running VM on the Hyper-V host. As an added bonus it associates each VM to its vmwp.exe's Process ID.

You can run this on your Hyper-V server, because it doesn't need a GUI.

The downside is that while I was working on this, I noticed that the performance counters were working great for a while, and then for no discernible reason they decided to all stay on zero. Well maybe it's not a bug, like Chris S says... but these counters might unfortunately not be very useful after all. Regardless, it would be very easy to modify the script to use the Virt. Storage Device counters instead.

The output looks like this:

PID     VMName               ReadBytesPerSec             WriteBytesPerSec
---     ------               ---------------             ----------------
5108    DC02                          483.90                            0
2796    DC01                               0                            0
3348    ECA01                     4782668.27                            0

#Requires -Version 3
function Get-VMPidAndIO
{
<#
.SYNOPSIS
    Gets the Process ID and I/O statistics of each virtual machine running on the Hyper-V host.
.DESCRIPTION
    Gets the Process ID and I/O statistics of each virtual machine running on the Hyper-V host.
    Currently only works for VMs using virtual IDE controllers.
    Requires Powershell 3 at a minimum.
.LINK
    http://myotherpcisacloud.com
.NOTES
    Written by Ryan Ries, June 2013.
    [email protected]
#>
    BEGIN
    {
        Try
        {
            $VMProcesses = Get-CimInstance -Query "Select ProcessId,CommandLine From Win32_Process Where Name ='vmwp.exe'" -ErrorAction Stop
        }
        Catch
        {
            Write-Error $_.Exception.Message
            Return
        }
    }
    PROCESS
    {

    }
    END
    {
        Foreach($_ In $VMProcesses) 
        {
            $VMName = $((Get-VM | Where Id -EQ $_.CommandLine.Split(' ')[-1]).Name)            
            [PSCustomObject]@{PID=$_.ProcessId;
                              VMName=$VMName; 
                              ReadBytesPerSec=[Math]::Round($(Get-Counter "\Hyper-V Virtual IDE Controller (Emulated)($VMName`:Ide Controller)\Read Bytes/sec").CounterSamples.CookedValue, 2);
                              WriteBytesPerSec=[Math]::Round($(Get-Counter "\Hyper-V Virtual IDE Controller (Emulated)($VMName`:Ide Controller)\Write Bytes/sec").CounterSamples.CookedValue, 2); }
        }

    }
}
4
  • That's not a bug, see comments in my answer.
    – Chris S
    Jun 3, 2013 at 13:03
  • Then I wonder why the counters seem to intermittently work? (Windows guests with full integration services.) And why they would choose the significantly less useful level of abstraction of per-VHD counters instead? :(
    – Ryan Ries
    Jun 3, 2013 at 13:10
  • Not sure why it would show anything intermittently. As for why, it's the way storage works in Hyper-V. Essentially there is only one storage controller for all VMs. By using an "IDE Controller" in the configuration you are just enabling IDE Emulation for the storage exposed to that VM (as opposed to the "SCSI Controller" that has nothing to do with SCSI, and just disables any sort of storage controller emulation). VMs that support Integration Services just skip the IDE emulation. The storage controller counters don't exactly care what VM is attached to each VHD, hence the way it works.
    – Chris S
    Jun 3, 2013 at 13:19
  • Awesome. Thanks for the info. I'm extremely interested in reading more about this, but I'm having a hard time turning up any good technical documentation. I think it would behoove MS to demystify a lot of this stuff for us. We'd be able to put more confidence in their solutions. I just want a Hyper-V Internals book. Is that too much to ask? :)
    – Ryan Ries
    Jun 3, 2013 at 13:37
1

Both of the other answers are useful. But I find that I get what you're looking for much more easily (when the VHDs are stored on local or locally shared disks instead of a file server) by opening up "Resource Monitor" and looking at the Disk tab. Look at "Disk Activity" and sort by "Total (B/sec)." You'll see the VHDs listed in order of their activity.

Obviously my strategy is a loose approximation for machines that you're locally logged into, which makes it less useful in a big, headless environment.

1
  • 1
    Jake, thank you for taking the time to respond. I'd really love to just look at the resource monitor, however, it is unavailable when the server is not running a full graphical install but only the Hyper-V edition or Server Core. Which is what I have.
    – the-wabbit
    May 31, 2013 at 18:02
1

I wrote a PowerShell GUI tool gathering Guest performance data for all VM's found on selected Hyper-V hosts. Data gathering and matching is done with get-counter and Hyper-V WMI information. Run from any member server, no modules required. I hope this will help in quickly troubleshooting performance problems on Hyper-V hosts and VM's.

Show Hyper-V Virtual Machine Guest Performance Statistics (PowerShell) https://gallery.technet.microsoft.com/Show-Hyper-V-Virtual-652fdd54

0

You may want to try a 3rd party tool that can help with detecintg virtual machine that is causing high I/O, and also has the ability to alert on such high I/O’s. You can check it out here - ApexSQL VM Monitor.

This tool can monitor large spectre of disk counters for host and virtual machine, and beside predefined counters, it can be set up to monitor and show graphical presentation of counters from Performance monitor.

Hope I helped.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .