4

Since upgrading some host servers to Hyper-V Core 2016, I've been having some problems with replication coupled with checkpoints.

In my present example, I have a replicated Hyper-V guest (Gen2, V5.0). We run a nightly backup process which will take a checkpoint of the replica, copy the VHDX files to a backup store then delete the checkpoint.

Some nights, the guest seems to get stuck applying replica changes and will not allow the checkpoint to be deleted. Our backup process will wait until the machine has finished applying changes, but in this case it never happens and the replica guest is stuck with "Applying Registered Delta..." in the Status column of Hyper-V Manager. Replication becomes critical and changes are no longer replicated as the backlog is too great.

Screenshot of Hyper-V Manager displaying the guest with "Apply Registered Delta..." as the status

Through the front-end I have the ability to right-click and select "Cancel Applying Replica Changes", but I have used this in the past and it has no effect other than removing the option from the menu. The only solution I've so far discovered is to completely reboot the host.

I have attempted to restart the VMMS service as similar issues in the past have resolved themselves after doing this, but in this case the service freezes with the status "Stopping" and can no longer be interacted with. I had to do a cold reboot to recover as the host got stuck doing a soft reboot.

I'm a bit at a wits end now as this has halted my upgrade plans until I can resolve this problem, and I'm unable to find a anyone posting about the same issue in Hyper-V 2016.

Other points of interest:

  • This is not specific to only one of the hosts or one of the guests
  • The issue will occur whether or not the guest is v5 or v8
  • This issue will occur whether or not the source host is Hyper-V 2012 or Hyper-V 2016
  • This issue did not/does not occur on Hyper-V Core 2012R2

  • There is plenty of disk space remaining (>600GB)

  • Replication is setup to keep 24 hourly checkpoints and changes are replicated every 30 seconds.
  • The host has several other running guests and replicated guests, but the issue has occurred on a host where it had only one replicated guest.
  • The latest windows updates have been applied.
  • This does mostly seem to happen on large guests which are likely to be replicating the most changes. I have had it occur on a smaller guest.

UPDATE

This issue has also occurred with "Applying Replica Changes..." in the status column, rather than the "Apply Registered Delta..." status.

UPDATE 16/06

I found one or two articles with similar issues which suggested Windows Defender was to blame. I uninstalled this from the server, but the issue has reoccurred after a couple of days.

UPDATE 20/06

I have found a forum post HERE describing a very similar issue. The problem was occurring when Windows Defender was disabled on a Windows Server 2016 installation. They ran the following tests:

  1. Windows 2016 Datacenter with latest updates on source and destination with >Trend Antivirus installed and NO Windows Defender - Replication breaks after >initial replication and causes Hyper-V Management service to go unresponsive.

  2. Windows 2016 Datacenter with latest updates on source and destination with NO >Trend and NO Windows Defender - Replication breaks after initial replication and >causes Hyper-V Management service to go unresponsive.

  3. Windows 2016 Datacenter with latest updates on source and destination with >Widows Defender and NO Trend - Replication works normally.

  4. Windows 2016 Datacenter with latest updates on source and destination with >Widows Defender and Trend Installed (Trend Disables Windows Defender) - >Replication breaks after initial replication and causes Hyper-V Management >service to go unresponsive.

I'm now wondering if perhaps my problem might be due to a key feature of Windows Defender not running or not being present on Hyper-V Core 2016. I'm quickly reaching the point where I will just have to downgrade from 2016 back to 2012.

UPDATE 4th July

I've begun the process of downgrading back to Hyper-V Core 2012 R2 as this whole situation was untenable. Hopefully Microsoft will sort this problem out in an update or the 2016 R2 release.

2
  • Update: I've disabled and uninstalled Windows Defender, and so far this has been running without any issue. It's too early to say this has fixed it, but things are looking good.
    – Kez Floyd
    Jun 15, 2017 at 11:03
  • 1
    This has happened again, so uninstalling Windows Defender did NOT solve the issue.
    – Kez Floyd
    Jun 16, 2017 at 9:12

1 Answer 1

2

I've managed to recover from this state several times how. This may not work if you've already tried to kill any of the services:

  • Download PSExec from Microsoft's Sysinternals website
  • Go to Task Manager -> Services tab.
  • Make a note of the PID numbers for 'vmms', 'Winmgmt' and 'WinRM'
  • Using a command window with Administrator access, launch PSExec with the command psexec.exe /i /s cmd.exe. This creates a new command window which is running under the "SYSTEM" account
  • In the new command window, run taskkill /f /pid #### command for each of the PIDs in the previous step.
  • Use the Task Manager -> Services tab to start all three services again. Skip any that have started themselves.
  • The "Applying Registered Delta" statuses will have disappeared from the VMs, and they should now be configurable.
2
  • I tried following this advice and I didn't see "Winmgmt" or "WinRM" running, and when I tried to kill "vmms" it said "SUCCESS: The process with PID X has been terminated." but vmms still appears in the task manager.
    – RyanTM
    Sep 27, 2021 at 20:55
  • 1
    Actually, maybe it did work! After killing it I tried running Get-Service vmms | Restart-Service in a powershell that I spawned from the system cmd.exe and that started vmms and cleared out the old one from the task manager eventually!
    – RyanTM
    Sep 27, 2021 at 21:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .