15

I have a Windows Server 2012 R2 virtual machine; yes with all the updates. Additional software include Microsoft SQL Server 2014 (was 2012 on a previous VM). The web hosting company includes xenpci.sys (EJBPV XenPCI Driver (Checked Build), James Harper) as part of their default installation on all VMs and Plesk.

Periodically, the OS either hangs, blue screens, or reboots. I do get mini dumps, though not all the time. The usual problem is:

Error: CRITICAL_STRUCTURE_CORRUPTION

The specific top level file, obviously not the cause, varies: win32k.sys, ntoskrnl.exe, xenpci.sys (the Xen driver, though only showed up a couple of times), and ndis.sys.

The OSR (Open System Resources) analyzer was not of much help. The WhoCrashed analyzer was a bit more helpful.

It stated:

17 crash dumps have been found and analyzed. Only 10 are included in this report. A third party driver has been identified to be causing system crashes on your computer. It is strongly suggested that you check for updates for these drivers on their company websites. Click on the links below to search with Google for updates for these drivers:

xenpci.sys (EJBPV XenPCI Driver (Checked Build), James Harper)

I tried to push the web hosting company to research the topic, but they can up empty handed. I am not convinced that the Xen drivers are at fault. WhoCrashed picked up on it, I presume merely because that was the last driver a couple of times and it is a third party, so that makes it guilty. I did not write WhoCrashed, so hard to comment further.

My question is how to troubleshoot the problem.

The web hosting company already tried giving me two new virtual machines throughout the last couple of years. The problem migrates. I installed SQL Server, but the OS and Plesk came by default. Okay, there is the mail server software too. The web hosting company also told me that they do not have other clients similarly complaining. They ran disk tests multiple times. Disk health is good.

I did not check the registry's health, but the problem goes across installations and happens pretty routinely, so I would have to discount that. I am on my third or fourth VM now.

Again, I mention Xen because WhoCrashed mentioned it, but I am not convinced that as the cause, and other clients really do use that. The system has adequate memory and storage, so that is not a problem.

UPDATE: Here are some answers from the web hosting company to my query.

In usual scenario, performance of the VM will get degraded once you uninstall the drivers. There might be some synchronization issues with the Hardware Node.

Am I using a checked or release build?

You are using a test-signed build, the same ones from developer's site.

How can I tell? The Xen PCI properties dialog in Device Manager did not say one way or the other. Is the entry in Device Manager the sole location? I checked in Programs and Features and saw nothing listed.

You can check the version under Add or Remove programs. Refer to the snapshot attached.

How/where I can I find where the latest version is on their site?

Developer's site is not working - http://www.meadowcourt.org/downloads/ You can donwload the latest signed releases from here - http://wiki.univention.de/index.php?title=Installing-signed-GPLPV-drivers

How can I tell which Xen, 0.11.0.373 belongs to (Xen 4.6? 3.0? x.y?)

We are using Xen 3.4.4, you can't see it from your VM. It can only be viewed from hardware node.

Update 2: The hosting company installed two James Harper software.

GPL PV Drivers for Windows
EJB PV Drivers for Windows
  • 2
    A quick Google search for "Xen Windows BSOD" turns out lots of results, especially when old Xen drivers are used on recent Windows VMs; the culprit is quite likely to be there. – Massimo Sep 8 '15 at 22:33
  • The easiest way to determine if the Xen driver is the culprit would be to uninstall it. – joeqwerty Sep 8 '15 at 22:57
  • I thought of that option, sadly I do not think my web hosting company will go for that. They told me on several occasions that the system needs the hypervisor to gain access to the PCIe slots and function correctly. I would have to search my huge message archive with them to get the exact wording, but basically they were not very willing. – Sarah Weinberger Sep 8 '15 at 23:16
  • 1
    Uninstalling the hypervisor drivers from a VM is very likely to achieve unpleasant results, up to not being able to boot anymore. – Massimo Sep 8 '15 at 23:17
  • 3
    @SarahWeinberger, when the web host gave you new VMs, were they on different hardware? A bunch of BSoDs showing different top level files that ultimately seem to be caused by the hypervisor driver sure seems like something that could be caused by bad memory. If your "new" VMs are on the same hardware, then it's possible this will keep manifesting. – briantist Sep 9 '15 at 2:56
18

xenpci.sys (EJBPV XenPCI Driver (Checked Build), James Harper)

(Checked Build) is a huge red flag. You absolutely should not be using "checked" builds of anything in production. If your hosting company loaded this driver for you, then they absolutely made a mistake.

Checked builds include extraneous symbols and extra error checking that aid developers. They are not production builds.

To further elaborate, what this tells me is that whatever error is causing the machine to stop probably still occurs in the un-checked build of the driver, however, it's probably just causing a non-fatal side effect such as a memory leak in the release build. But in the checked build, because of the more stringent error checking, it stops the entire OS. That's the point of checked builds, to accentuate errors and shove them in the developer's faces, before they ship the code to customers.

To further elaborate, it doesn't really matter if other VMs also have that same exact driver loaded (the checked build) and don't seem to be crashing. Some component specific to that VM is invoking some certain behavior or state that is triggering the bug in that driver. (Drivers and applications interact in all sorts of ways, maybe two machines have the same buggy driver loaded, but only one of the servers has SQL installed, and since the server has SQL installed it does this unique memory page locking in a way that the other server doesn't do, which causes the 3rd party driver bug to rear its ugly head. (Just an example.))

There's really no where else to place the blame here. You cannot run checked builds of drivers in production and expect to have a good time. They're only for development and testing purposes.

Lastly, the only other place to go from here would be to collect full dump and run it through WinDBG. You can spend six hours of intense debugging, unwinding stacks, tracing threads, following IRPs to their completion ports... or you can just get rid of that checked build driver. :)

Might also try running the driver through Driver Verifier. In a test environment. Where checked builds should stay. ;)

  • 1
    I will ask about the hosting company if I am using a checked build or not. I just checked Device Manager and all it says for "Xen PCI Device Driver" is "9/17/2014 | 0.11.0.373". I do not see checked or release anywhere in the properties dialog, but the hosting company will know more. Stay tuned. – Sarah Weinberger Sep 8 '15 at 23:23
  • 2
    Then where did you get the "Checked Build" from in your post? – Ryan Ries Sep 8 '15 at 23:44
  • I copy and pasted the WhoCrashed analysis summary block at the bottom of the report. OSR's report was not very helpful, which is why I focused on the WhoCrashed one, but the problem may not be Xen related and could be something else. At one point, the one of the web hosting company support agents tried to pin the blame on Microsoft and IIS, for which I dismissed that out of hand. I agree with WhoCRashed in suspecting a third party agent. – Sarah Weinberger Sep 8 '15 at 23:50
  • @SarahWeinberger Check the file size, and at least compare between the servers you have access to; I don't know for certain, but the file sizes should be different for checked and non-checked builds (the differences are significant enough that it should show up in file size). If all else fails, compute a checksum; even MD5 will do just fine here. The checked and non-checked builds will be different there, even if the file version is the same. – a CVn Sep 9 '15 at 7:16
  • @MichaelKjörling I have to go to the site and understand the downloads. Up until the host's post, I thought the download came from a different site, the one that I posted originally. My guess is that the site has one download, a released version, but checked builds can be digitally signed too. Sadly, the uninstall is on the hardware node, which seems to be an older version of Windows, as he talks "Add/Remove Programs" not "Programs and Features". Either way, uninstall is on their end. I already checked P&F and did not see Xen. – Sarah Weinberger Sep 9 '15 at 12:03
7

Xen 3.4.4 is too old. 13 March 2013

Windows 2012 R2 got released on 18 October 2013.

To give you an actual comparison, XenServer from Citrix added Windows Server 2012 R2's support in their version 6.2SP1, which was released the 13 December 2013. (http://support.citrix.com/article/CTX139788)

See that for the GPLPV's driver; you see only reference to Xen branch 4.4.0 for 2008R2 ..

The signed drivers from ejbdigital work great on Xen 4.4.0. If you experience a bluescreen while installing these drivers, or after a reboot after installing them, please try adding device_model_version = "qemu-xen-traditional". I had an existing 2008 R2 x64 system that consitently failed with a BSOD after the gpl_pv installation. Switching to the 'qemu-xen-traditional' device model resolved the issue. However, on a clean 2008 R2 x64 system, I did not have to make this change, so please bear this in mind if you run into trouble. http://wiki.xen.org/wiki/Xen_Windows_GplPv

See that for the official statement of xen project on Windows's support. I don't mean it can't run it, but you see the support. You will run a production server on that platform ?

Does Xen Project support Microsoft Windows?

The paravirtualized approach we use to get such high performance has not been usable directly for Windows to date. However Xen 3.0 added Intel VT-x support to enable the running of unmodified guest operating systems, including Windows XP & 2003 Server, using hardware virtualization technology. Xen 3.0.2 and later support AMD Pacifica technology as well. Check to see if your CPU is among the list of HVM Compatible Processors, and if your motherboard is among the list of HVM Compatible Motherboards.

(Note: This does not necessarily mean, that just about any OS runs in HVM Mode! Reports vary about getting *BSD to work in HVM mode, on the Mailing List there's only 1 success for OpenBSD (http://www.openbsd-france.org/ml/archives/msg02494.html), no success message for the others, but some problem reports) http://wiki.xenproject.org/wiki/Xen_FAQ_Drivers,_Windows

  • 1
    I marked the answer up (I am #4) and forwarded the answer along with this URL to my web hosting company. I never even heard of Xen prior to this problem, so I am new to the whole thing. Your point on version dates is very valid. How could Xen 3.4.0 support Windows 2012R2, when that was not even out at the time. I shall see what the hosting company says. Sadly, they ruled out Microsoft Hyper-V, not sure why. – Sarah Weinberger Sep 11 '15 at 14:16
  • 1
    This answer made a difference. The web hosting company, infamous for changing nothing, just wrote me this email: "Please be informed that we are planning to update the version of Xen server on which your cloud server is running, to the latest since the older versions seems to be vulnerable." Wow! – Sarah Weinberger Sep 12 '15 at 13:49
  • Thanks for the feedback. Iam glad if my answer helped :) – yagmoth555 - GoFundMe Monica Sep 12 '15 at 16:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.