4

I have a Windows Server 2012R2 hyper-v host. It is running a number of VMs, including a few linux VMs.

I have one linux VM that has been occationally, but somewhat regularly crashing somewhere in the kernel, resulting in the VM becoming completely unresponsive.

I know hyper-v has a "heartbeat" facility, and it successfully detects the VM crashing as a loss-of-heartbeat.
However, I'm at a loss as to how to set up hyper-v so that it actually does anything with the heartbeat signal. Apparently at one point, there was the option to reset the VM if a heartbeat is lost for > 1 minute, but it's either been removed, or placed somewhere I cannot find.

I only have this one server, so doing things like clustering or fail-over are not really viable (this is a home lab server).


Before anyone comments, Yes, I am trying to track down the cause of the crash in the VM, but it's irregularity has been making it difficult.

3
  • AFAIK, VM monitoring is only available for Windows Server 2012 and Windows Server 2012 R2 virtual machines.
    – joeqwerty
    Oct 17, 2014 at 16:27
  • @joeqwerty - The heartbeat monitoring works fine on linux. The "Heartbeat OK/Heartbeat Lost" status indicator is fully functional. The linux support of hyper-v is actually quite good. There are paravirtualized drivers for all the base functions, full guest support, and the latest kernel versions even support memory ballooning.
    – Fake Name
    Oct 17, 2014 at 23:23
  • Unless I'm missing what you mean (it's just disabled for non Server 2012 VMs?), guest additions/heartbeat/memory balloning work on Windows 7 as well.
    – Fake Name
    Oct 17, 2014 at 23:25

2 Answers 2

2

powershell is your rescue:

$VM = Get-VMIntegrationService -VMName test-vm -Name Heartbeat
if ($VM.PrimaryStatusDescription -ne "OK")
{
    write-host "VM Dead ? restarting ..."
    Stop-VM test-vm -Force -TurnOff
    Start-VM test-vm
}

just add this to task scheduler.

If You VM does not support heartbeat then modify this to ping the VM instead of checking the HB.

0

I needed a solution for monitoring the heartbeat of all Hyper-V virtual machines and automatically preforming a hard reset when the VM locked up. As you may already know. it seems as though you can do this using Hyper-V when you have a cluster setup, but with only a single Hyper-V host, that is not possible without some custom scripting.

I found a nice VBS the monitors the heartbeat of all systems, and even has a configurable retry count and grace period in the latest version. The original script and description can be found here. It'd be a good idea to schedule this to run with system startup (don't just place it in the startup folder, because that would require the Hyper-V Administrator to login to the system before the script would start running).

Usage: - Copy VMHeartBeat.vbs script to a location of your choice - Either double click on the script or open a command prompt,change to the folder where the script is copied and then run cscript.exe VMHeartBeat.vbs

Note: If you double click on the script wscript.exe engine or whichever is your default scripting engine will be used to run the script. When you use cscript.exe, you need to keep the command window always open.

This script will create a log file at the same location as the script and keep appending the success/failure events to it. Log file name is VMHeartBeatEvents.log

That is it. You are all set to detect the VM heart beat loss and then perform a hard reset.

To test if the script is working or not, you can download NotMyFault.exe and then copy it to the VM. Now, just run VMHearBeat.vbs on Hyper-V host and open up notmyfault.exe inside the VM, select a bug and click “Do Bug”. Based on what kind of a bug you select, VM may blue screen. At this point in time, VMHearBeat.vbs would have detected the heart beat loss and performed a hard reset of the virtual machine. You may check the log file, as mentioned earlier, to see if the hard reset was successful or not.

For the sake of preserving the script, I've also pasted the latest version of the .VBS code (with retry count and grace period) below.

UPDATE: The latest version of the script actually had some bugs, so the script below has been modified and I confirmed it is working on Server 2012R2. \v2 needed to be added to line #20 and 10 needed to be changed to 11 on line #4 of the VMHardReset function.

Option Explicit

'Define Event log constants here
Const EVENT_INFO = 4
Const EVENT_ERROR = 1
Const RETRY_COUNT = 3
Const CHECK_EVERY_X_SECS = 6

Dim objSummaryInfo, objWMIService, objVMService, intValArray, objVMitems
Dim InParam, OutParam, objItem, strQuery
Dim Job, objShell
Dim colArgs, strLog, dict, compName

Set colArgs = WScript.Arguments.Named

'Get Command Line arguments in to some variables
strLog = colArgs.Item("log")

Set objShell = Wscript.CreateObject("Wscript.Shell")
Set objWMIService = GetObject("winmgmts:\\" & "." & "\root\virtualization\v2")
Set objVMService = objWMIService.ExecQuery("SELECT * FROM Msvm_VirtualSystemManagementService").ItemIndex(0)
intValArray = Array(0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199)

Set dict = CreateObject("Scripting.Dictionary")

While (1)
    Wscript.echo(CStr(Now()) +": Testing Loop ")   
    objVMItems = objVMService.GetSummaryInformation(NULL,intValArray,objSummaryInfo)
    for each objItem in objSummaryInfo
        compName = objItem.ElementName
        if (not dict.Exists(compName)) then
            dict.add compName,0
        end if

        If (objItem.EnabledState=2 AND (objItem.HeartBeat = 6 OR objItem.HeartBeat = 13)) Then
            dict.Item(compName)=dict.Item(compName)+1

            WriteLog("    **"+compName+"** missed heartbeat check; count "+CStr(dict.Item(compName)))

            if (dict.Item(compName) >= RETRY_COUNT ) then       
                dict.Item(compName)= 0
                Wscript.echo(CStr(Now()) +": Now performing a hard reset on "+compName)
                VMHardReset(compName)
            end if
        Else
            dict.Item(compName)= 0
        End If
    Next

    Set objVMItems=Nothing
    Set objItem = Nothing
    Set objSummaryInfo = Nothing

    Wscript.Sleep CHECK_EVERY_X_SECS * 1000
Wend

Set objWMIService = Nothing
Set objVMService = Nothing
Set InValArray = Nothing

Function VMHardReset(vmElementName)
    Dim objvm, strLine
    Set objvm = GetComputerSystem(vmElementName)
    Set InParam = objvm.Methods_("RequestStateChange").InParameters.SpawnInstance_()
    InParam.RequestedState=11
    Set OutParam = objvm.ExecMethod_("RequestStateChange",InParam)
    If (TrackJob(OutParam)) Then
        strLine = "Virtual Machine: " & vmElementName & " has been successfully recovered from a hearbeat failure"
        'Write Success Event to text log
        WriteLog(strLine)
        Set strLine=Nothing
        'Write a Windows Event Log
        objShell.LogEvent EVENT_INFO, strLine
    Else
        strLine = "Virtual Machine: " & vmElementName & " could not be recovered from a hearbeat failure"
        'Write Failure Event to text log
        WriteLog(strLine)
        Set strLine=Nothing
        'Write a Windows Event Log
        objShell.LogEvent EVENT_ERROR, strLine

    End If
    Wscript.echo(CStr(Now()) +": "+strLine)
    Set InParam = nothing
    Set objvm = nothing
    Set vmElementName = Nothing
End Function


Function GetComputerSystem(vmElementName)
    strQuery = "select * from Msvm_ComputerSystem where ElementName = '" & vmElementName & "'"
    set GetComputerSystem = objWMIService.ExecQuery(strQuery).ItemIndex(0)
    'Set GetComputerSystem = Nothing
    Set strQuery = Nothing
End Function


Function TrackJob(OutParam)
    Dim Job
    If (OutParam.ReturnValue = 0) Then
        TrackJob=True
    ElseIf (OutParam.ReturnValue <> 4096) Then
        TrackJob=False
    Else
        Set Job = objWMIService.Get(OutParam.Job)
        While (Job.JobState = 3) or (Job.JobState = 4)
            Set Job = objWMIService.Get(OutParam.Job)
        Wend

        If (Job.JobState <> 7) Then
            TrackJob=False
        Else
            TrackJob = True
        End If
    End If
    Set OutParam = Nothing
    Set Job = Nothing
End Function


Sub WriteLog(line)
    Dim objFileStream, objFileSystem
    Set objFileSystem = CreateObject("Scripting.FileSystemObject")
    Set objFileStream = objFileSystem.OpenTextFile("VMHeartBeatEvents.log", 8, true, -1)
    objfileStream.WriteLine CStr(Now()) +": "+line
    objfileStream.Close
    Set objFileSystem = Nothing
End Sub
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .