2

We have a Windows Server 2003 box running as a web server in a remote data centre. Last night I installed a large batch of Windows Updates and then triggered a reboot around 1:30am via RDP. The reboot failed and although the server is still running, we're locked out via RDP. Cue panic and mayhem!

I did not route the applications to another machine (which is only partly possible with the equipment we have currently) because a few hours' down time for upgrades in the middle of the night is acceptable to our customers.

For now, I am liaising with the data centre staff to reboot the machine via other means late tonight, but obviously I want to avoid this in the future.

My questions:

  • How can I prevent losses of service like these? Note that we are a very small company with a light load on our server, so while I am interested in best-practices that involve buying lots of extra hardware, I would prefer to hear about cheaper things that can be done.
  • Is having RDP as the only means to perform certain important tasks (namely modifying the database to solve customer service issues) a liability?

Details of the machine:

  • Windows Server 2003, unsure which service pack
  • Running two websites on ASP.NET 3.5, and another four on ASP classic
  • SQL Server 2005 backend for all the websites
  • Uses a continuous backup solution to an identically configured machine that is ready to be plugged in and switched on
  • Running a VMware instance that contains a staging environment and is not mission critical
  • One hard drive partition with mirroring
  • 4GB RAM
  • Core 2 Duo ~ 2Ghz

Thanks in advance. More information on request.

UPDATE:

Some excellent answers here so far.

For remote management, several have suggested using KVM and remote power management over IP, or hardware such as HP's iLO or Dell's DRAC. We have HP servers so I will look into iLO. Irritatingly, our hosting centre has KVM over IP for all its machines but don't allow access to customers as it's not set up securely. In selecting future hosting services, I will make sure this is not the case.

For prevention, mh suggested stopping services and closing sessions that may be preventing the reboot. In our case that would probably have identified the issue and prevented the problem. It seems like the VMware instance running our staging environment was not shut down and that stopped the main server restarting.

John Gardeniers suggested performing reboots manually after installing updates, and not letting Windows Update carry out the reboot. I will do this in future.

Thanks everyone.

Improve this question

9 Answers 9

Reset to default
4

There are several options for accessing the console remotely without relying on RDP into a working Windows install:

  1. Lights Out Management - some servers from Sun, HP, IBM and several others have a Lights Out Management Chip (LOM). I have much of my experience with HP's iLO technology which has saved me several trips. Essentially this chip gives you authenticated remote access to the controls on the front of the server, and most instances much much more. The down side is if you server dosn't have it you can't use it.

  2. IP KVM's or Remote Power Management - several vendors provide products that allow you to access either the keyboard, video and mouse remotely (IP KVM) or less expensivly the ability to manage power remotely by cycling the mains power to the server off and on again via a PDU. The latter option is not recommended unless you are sure simply power cycling the server will help.

The final option would be to not install updates over night, and schedule short outages during the day when you or your "hands and eyes" can be at the data center to sort out any problems. This is at the whim of your customer really, although a quaterly downtime is often a good thing to build in to your agreements.

Improve this answer
2
  • +1 for mentioning IP KVMs and remote power management, and +1 again (if I could) for mentioning scheduled maintenance windows.
    – Evan Anderson
    Jul 3, 2009 at 11:11
  • Some data centres provide an IP KVM/remote power management as part of their service too
    – Dan
    Jul 3, 2009 at 11:17
1

If your server is a Dell or HP you would be able to do remote hard resets using the BMC (Dell) or ILO (HP) controller. Both Dell and HP have a more expensive option that gives you a remote keyboard and screen. With the Dell you install a DRAC card, and with the HPs I think you just have to install a licence key to enable the extra functionality.

If the server isn't a Dell or HP you're stuck. Some other servers have similar remote controllers but many don't. Servers do occasionally hand when restarting, often for no reason that I can see. The only other option is to use the server just for running VMs, make sure it doesn't see the Internet, and never update it (or only under controlled conditions). If a VM fails to restart cleanly it's easy to restart it from the host.

JR

Improve this answer
2
  • Your VM-based solution sounds interesting. Have you had much experience with it? Any caveats to be aware of?
    – Alex
    Jul 3, 2009 at 13:38
  • @Alex: I use Windows 2008 and Hyper-V and it works brilliantly well (I'm sure VMWare would be just as good). There are no caveats beyond the obvious e.g. you need a Windows licence for every VM beyond the first and this can get expensive. I wouldn't virtualise a big SQL Server (for light use SQL Server works fine on Hyper-V) but web servers virtualise vey well.
    – John Rennie
    Jul 3, 2009 at 15:29
1

That reboot, was it a part of applying the patches? I've had several occasions where allowing the update process to perform the reboot has caused the machine to end up not fully shutting down, just as you described. In most cases all that was required was a key press or movement of the mouse, although there was no prompt on the screen. Unfortunately, it always happened when I was away from the office. These days I use a script to install the updates but not allow a reboot at the end. The reboot is performed separately. I've had no further trouble since.

Improve this answer
1

I've had the same problem before in our data centre, fortunately I had an iLO card in my HP servers and a DRAC in my Dell box, however one of the boxes it happened on did not have any remote management. the only thing I could do was phone up the data centre and ask if one of their guys could power down and power up for me.

Improve this answer
1

For server reboots, the first thing I always do is ensure that I'm present during the reboot process. If I can't be physically present, at least I like to be able to ping -t it so that I can observe it going down and coming back up.

Before rebooting I stop services. As many as possible, ensuring that only the most basic list of services are up and running at the time it goes down. This includes any websites, email, AV, security agents, anything like that. Most of the "hang while going down" cases I've seen can be directly attributed to a service getting stuck in the "stopping" state during the reboot, and this avoids it. This is the single most useful thing to do, and unfortunately there is no definitive applies-everywhere list of services to stop; experience and your own knownledge of what's on the server are required here.

Then I'll check for any other RDP sessions that may still be hanging around. A disconnected but not logged off session may still have a user profile loaded. Also check for any other connections to the server that may be active.

Then I reboot, and during the reboot I'll fire up a command-prompt and ping -t it. I normally don't need to see what's on-screen at this stage, although it can be useful if you have the necessary prerequisites.

Occasionally if a server hangs during the reboot, I've been able to get at it via Computer Management, and - as I said above - find a service stuck in the "Stopping" state. A remote "kill" utility can come in handy here.

Improve this answer
1

Most of the time we would reboot as soon as the patches are applied. Pushing patches and waiting for a more opportune moment to reboot has only caused problems. All of the others have provided excellent, and tried and true, methods to reboot a wayward box. ILOs and DRACs are great, using the PSTools shutdown tool is a good idea too. I've found that sometimes the normal methods failed but using something like VNC or Dameware allowed me to log into the box and kick it over.

Improve this answer
0

If you had 2 remote servers (and assuming they're able to talk to each other), on the other server you could run

Shutdown /m \\Server /r /c "Remote Restart" /t 30

(For Win2000 there was RCMD (part of the resource kit) which saved me a couple of times but I don't think that's an option any more).

Improve this answer
2
0

One thing not mentioned, but which was useful for me in at least two cases I remember, is WakeOnLAN - if you have at least two servers in the same LAN in your data center.

See this question here on Serverfault.com: Wake on LAN tool for Windows

Improve this answer
0

We normally use Windows Remote Desktop for remote administration, but we have also have Radmin (http://www.radmin.com/) on most of our remote servers. On rare occasion a system will be available via Radmin, but not RDP. This can be a life saver. You might try VNC also. You will have to consider the security aspects of either of these of course.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .