1

Summary: A long-running virtual Windows Server 2012 system this weekend became very slow; from remote we did Restart-Computer -ComputerName xyzzy -Force, which succeeded by all accounts; except that some processes look like they have survived at least 10 reboots.

And as this is the first time I see this in 20+ years of using Windows, I'd really like to know what happened.


Here is a chronological story:

  • This system has been rebooted at least 10 times between August 8, 2014 and January 19, 2015. The last reboot was on January 16.
  • The VM, which is running on VMware ESXi 5.5.0, was barely reachable anymore through web-based UIs and Remote Desktop: its behavior suggested that some process or processes were taking up the CPU.
  • We decided to reboot the server from another Windows system, using Powershell: Restart-Computer -ComputerName xyzzy.
  • This took quite a while and then returned an error indicating that users were still logged in.
  • Fair enough, let's apply force: Restart-Computer -ComputerName xyzzy -Force.
  • This took quite a while and then returned.
  • In the meantime I had a ping -t xyzzy running, to check that the network interface would go down, to check that the reboot really occurred. It did: for about a minute or two that repeatedly gave me Request timed out. and then the ping replies resumed.
  • Then I successfully connected with Remote Desktop. This showed a dialog saying that the system shut down unexpectedly, and asked me to fill in the reason.
    • This was a bit unexpected, but may be standard after a remote Restart-Computer.
  • To find out the reason for that dialog I checked the event viewer, and could find nothing amiss. And I did observe all kinds of Kernel-Boot and lots related events, reporting uptimes of 20 seconds and 79 seconds.
  • (nothing for a while)
  • Then we observed that the system was becoming slow again, and I connected with Remote Desktop and had a look with Process Explorer. That resulted in the following:

    Process Explorer screenshot

    The right-most column, with dates of 1/19 (January 19th, 2015) and 8/14 (August 14th, 2014), is the 'Start Time' column of Process Explorer, and I sorted on that column.

  • That August 2014 sounds like a realistic date of when we last rebooted this system This same date-time was also observed by another process which manages the shown java.exe processes: it reported processes started 158 days ago. And according to Process Explorer, these were some of the processes that were eating up CPU and making the system slow. Finally, the per-process log files for the java.exe processes 3248, 4568, and 4236 give no indication whatsoever that these processes were ever stopped: they have log messages from both August 14th, 2014, and from the first minutes after the reboot.

So I can come to only one conclusion: the system has rebooted for about 90%, including the network stack and something that writes Kernel-Boot events, but 10 processes survived the reboot.

So, what could reasonably have caused this?

Was there perhaps a weird mixup of an old August 2014 VMware image/snapshot, and a fresh boot? Note that VMware has no snapshot for this system.


Note that since the system was becoming unusable again, I did a Restart-Computer -Force locally, causing a reboot, and this time there were no processes left over. So everything looks fine now, but I don't have the live system anymore: I can only do forensics.

0

You must log in to answer this question.