Since upgrading some host servers to Hyper-V Core 2016, I've been having some problems with replication coupled with checkpoints.
In my present example, I have a replicated Hyper-V guest (Gen2, V5.0). We run a nightly backup process which will take a checkpoint of the replica, copy the VHDX files to a backup store then delete the checkpoint.
Some nights, the guest seems to get stuck applying replica changes and will not allow the checkpoint to be deleted. Our backup process will wait until the machine has finished applying changes, but in this case it never happens and the replica guest is stuck with "Applying Registered Delta..." in the Status column of Hyper-V Manager. Replication becomes critical and changes are no longer replicated as the backlog is too great.
Through the front-end I have the ability to right-click and select "Cancel Applying Replica Changes", but I have used this in the past and it has no effect other than removing the option from the menu. The only solution I've so far discovered is to completely reboot the host.
I have attempted to restart the VMMS service as similar issues in the past have resolved themselves after doing this, but in this case the service freezes with the status "Stopping" and can no longer be interacted with. I had to do a cold reboot to recover as the host got stuck doing a soft reboot.
I'm a bit at a wits end now as this has halted my upgrade plans until I can resolve this problem, and I'm unable to find a anyone posting about the same issue in Hyper-V 2016.
Other points of interest:
- This is not specific to only one of the hosts or one of the guests
- The issue will occur whether or not the guest is v5 or v8
- This issue will occur whether or not the source host is Hyper-V 2012 or Hyper-V 2016
This issue did not/does not occur on Hyper-V Core 2012R2
There is plenty of disk space remaining (>600GB)
- Replication is setup to keep 24 hourly checkpoints and changes are replicated every 30 seconds.
- The host has several other running guests and replicated guests, but the issue has occurred on a host where it had only one replicated guest.
- The latest windows updates have been applied.
- This does mostly seem to happen on large guests which are likely to be replicating the most changes. I have had it occur on a smaller guest.
This issue has also occurred with "Applying Replica Changes..." in the status column, rather than the "Apply Registered Delta..." status.
I found one or two articles with similar issues which suggested Windows Defender was to blame. I uninstalled this from the server, but the issue has reoccurred after a couple of days.
I have found a forum post HERE describing a very similar issue. The problem was occurring when Windows Defender was disabled on a Windows Server 2016 installation. They ran the following tests:
Windows 2016 Datacenter with latest updates on source and destination with >Trend Antivirus installed and NO Windows Defender - Replication breaks after >initial replication and causes Hyper-V Management service to go unresponsive.
Windows 2016 Datacenter with latest updates on source and destination with NO >Trend and NO Windows Defender - Replication breaks after initial replication and >causes Hyper-V Management service to go unresponsive.
Windows 2016 Datacenter with latest updates on source and destination with >Widows Defender and NO Trend - Replication works normally.
Windows 2016 Datacenter with latest updates on source and destination with >Widows Defender and Trend Installed (Trend Disables Windows Defender) - >Replication breaks after initial replication and causes Hyper-V Management >service to go unresponsive.
I'm now wondering if perhaps my problem might be due to a key feature of Windows Defender not running or not being present on Hyper-V Core 2016. I'm quickly reaching the point where I will just have to downgrade from 2016 back to 2012.
UPDATE 4th July
I've begun the process of downgrading back to Hyper-V Core 2012 R2 as this whole situation was untenable. Hopefully Microsoft will sort this problem out in an update or the 2016 R2 release.