Summary: A long-running virtual Windows Server 2012 system this weekend became very slow; from remote we did Restart-Computer -ComputerName xyzzy -Force, which succeeded by all accounts; except that some processes look like they have survived at least 10 reboots.

And as this is the first time I see this in 20+ years of using Windows, I'd really like to know what happened.

Here is a chronological story:

  • This system has been rebooted at least 10 times between August 8, 2014 and January 19, 2015. The last reboot was on January 16.
  • The VM, which is running on VMware ESXi 5.5.0, was barely reachable anymore through web-based UIs and Remote Desktop: its behavior suggested that some process or processes were taking up the CPU.
  • We decided to reboot the server from another Windows system, using Powershell: Restart-Computer -ComputerName xyzzy.
  • This took quite a while and then returned an error indicating that users were still logged in.
  • Fair enough, let's apply force: Restart-Computer -ComputerName xyzzy -Force.
  • This took quite a while and then returned.
  • In the meantime I had a ping -t xyzzy running, to check that the network interface would go down, to check that the reboot really occurred. It did: for about a minute or two that repeatedly gave me Request timed out. and then the ping replies resumed.
  • Then I successfully connected with Remote Desktop. This showed a dialog saying that the system shut down unexpectedly, and asked me to fill in the reason.
    • This was a bit unexpected, but may be standard after a remote Restart-Computer.
  • To find out the reason for that dialog I checked the event viewer, and could find nothing amiss. And I did observe all kinds of Kernel-Boot and lots related events, reporting uptimes of 20 seconds and 79 seconds.
  • (nothing for a while)
  • Then we observed that the system was becoming slow again, and I connected with Remote Desktop and had a look with Process Explorer. That resulted in the following:

    Process Explorer screenshot

    The right-most column, with dates of 1/19 (January 19th, 2015) and 8/14 (August 14th, 2014), is the 'Start Time' column of Process Explorer, and I sorted on that column.

  • That August 2014 sounds like a realistic date of when we last rebooted this system This same date-time was also observed by another process which manages the shown java.exe processes: it reported processes started 158 days ago. And according to Process Explorer, these were some of the processes that were eating up CPU and making the system slow. Finally, the per-process log files for the java.exe processes 3248, 4568, and 4236 give no indication whatsoever that these processes were ever stopped: they have log messages from both August 14th, 2014, and from the first minutes after the reboot.

So I can come to only one conclusion: the system has rebooted for about 90%, including the network stack and something that writes Kernel-Boot events, but 10 processes survived the reboot.

So, what could reasonably have caused this?

Was there perhaps a weird mixup of an old August 2014 VMware image/snapshot, and a fresh boot? Note that VMware has no snapshot for this system.

Note that since the system was becoming unusable again, I did a Restart-Computer -Force locally, causing a reboot, and this time there were no processes left over. So everything looks fine now, but I don't have the live system anymore: I can only do forensics.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.