A service tech was on-site Friday, and among other things, installed a second RAID array (RAID 1) for user data in this Windows 2003 server. It boots from a RAID 5 array. This server is the only server on this network, and it's also the primary domain controller.
When he brought the server back up after installing hardware, it reported one drive on the RAID 5 array as failed. We talked about it, and decided the safest thing to do was to take a disk image to a USB hard drive, then to open her up to see whether a connection had simply been missed, or whether the drive had indeed failed.
I installed a trial version of Acronis True Image. I took a sector-by-sector backup of the RAID 5 array, and that finished with one or two errors. Since I had space on the USB drive, I also took a file-level backup of the RAID 5 data, also with Acronis True Image.
Since I also needed a regular backup, I started a full backup at midnight last night using NTBackup, writing to a NAS device. (The tape drive is a long story, and not relevant at the moment.) This backup should have taken at least 8 hours.
When I checked it this morning, I saw only a gray screen. It has a responsive mouse cursor on it, but nothing else. The server isn't hung; I can see the drives from other computers on the network, I can create and delete files, I can run server-based accounting software from the RAID 5 array and access the databases that are on the RAID 5 array. NTBackup finished in about 3 hours with 33 GB written to NAS--that's almost certainly a failed backup.
I can't shutdown blind from the console, because I have no idea now what's on the screen. I tried to shutdown from another computer using psexec and also using Powershell. Both those tries failed with security errors. (I don't think you can shutdown a domain controller that way, but I'm not sure.)
I do not want to just hit the power switch on a running server that reports a failed drive. But is that the only alternative I have left?
Finally had to quiet the server and kill the power. That brought everything back to the state it was in before the gray screen. The system log shows that an automatic Windows update restarted the server; that update also killed the NTBackup.