For me what stopped my install, was the vmware video card adapter being apparently incompatible with windows 10. There is an HTML file in the panther directory on the machine (/c$/$WINDOWS.~BT/Sources/Panther/ActionableReport.html) where it displays the error "this device isn't compatible in windows 10. contact the manufacturer for more info" and "VMware SVGA 3d (you'll have a problem with your display in windows 10)".
Vmware svga 3d error
I guess server core does not have the ability to display html files, and this causes the blank blue window like above from the OP.
It was fixed by re running setup.exe as "setup.exe /compat ignorewarning", after mounting the datastore ISO. This allows the install to complete in my case and bypass the vmware error. Note also that you need to have a slash before the ignorewarning command.
Check the panther directory for logs to as exactly why the install fails if you have a similar problem. This particular bug has been around on search engines since 2015, but mostly seemed to affect people who were trying to upgrade windows 10. Hard to believe it is still a bug, as pretty much everyone would be using vmware, its not some fly by night company or driver.
Another solution i saw mentioned is to change the video driver to generic microsoft VGA adapter, which apparently fixed it too.