We have a Windows Server 2003 box running as a web server in a remote data centre. Last night I installed a large batch of Windows Updates and then triggered a reboot around 1:30am via RDP. The reboot failed and although the server is still running, we're locked out via RDP. Cue panic and mayhem!
I did not route the applications to another machine (which is only partly possible with the equipment we have currently) because a few hours' down time for upgrades in the middle of the night is acceptable to our customers.
For now, I am liaising with the data centre staff to reboot the machine via other means late tonight, but obviously I want to avoid this in the future.
My questions:
- How can I prevent losses of service like these? Note that we are a very small company with a light load on our server, so while I am interested in best-practices that involve buying lots of extra hardware, I would prefer to hear about cheaper things that can be done.
- Is having RDP as the only means to perform certain important tasks (namely modifying the database to solve customer service issues) a liability?
Details of the machine:
- Windows Server 2003, unsure which service pack
- Running two websites on ASP.NET 3.5, and another four on ASP classic
- SQL Server 2005 backend for all the websites
- Uses a continuous backup solution to an identically configured machine that is ready to be plugged in and switched on
- Running a VMware instance that contains a staging environment and is not mission critical
- One hard drive partition with mirroring
- 4GB RAM
- Core 2 Duo ~ 2Ghz
Thanks in advance. More information on request.
UPDATE:
Some excellent answers here so far.
For remote management, several have suggested using KVM and remote power management over IP, or hardware such as HP's iLO or Dell's DRAC. We have HP servers so I will look into iLO. Irritatingly, our hosting centre has KVM over IP for all its machines but don't allow access to customers as it's not set up securely. In selecting future hosting services, I will make sure this is not the case.
For prevention, mh suggested stopping services and closing sessions that may be preventing the reboot. In our case that would probably have identified the issue and prevented the problem. It seems like the VMware instance running our staging environment was not shut down and that stopped the main server restarting.
John Gardeniers suggested performing reboots manually after installing updates, and not letting Windows Update carry out the reboot. I will do this in future.
Thanks everyone.