Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We set up a new server here a few weeks ago that I am informally responsible for managing.

Almost everything works perfectly except for one thing: Every so often it hangs without warning.

Some facts about this hang:

  • It is not a single application or service; the entire system is non-responsive.
  • Nothing is displayed (monitor acts as though there's no VGA signal).
  • The power LED is on and the fans are running.
  • Pressing the power button does nothing (normally it would shut the machine down).
  • Pings generally time out; once it did respond, another time I got "destination host unreachable".
  • Event logs show nothing (literally nothing at all) from before the hang until the hard reboot.
  • There are no performance problems, strange errors, or other obvious signs of impending doom leading up to the eventual hang.
  • The machine is generally not heavily loaded (it's for development, not production), and the hangs appear to be occurring at non-peak times of day (between midnight and 6 AM).

Some additional facts about the machine/environment:

  • Windows Server 2008 R2
  • Running SQL Server 2008 and IIS (not much else)
  • All drivers up to date, patches installed, etc.
  • No vendor-supplied diagnostics (not "top tier").
  • The machine is completely new, not merely reformatted or repurposed. No recent changes although the machine is less than a month old to start with.

I don't expect any easy answers here. What I'd like to know his I can methodically determine the root cause of this problem, be it a misbehaving service, defective hardware, or something else.

Is there any kind of logging I can set up that will help me get to the bottom of this? Any hardware diagnostics or remote monitoring? Anything else I can do to help me discover what's actually happening, or at least be able to eliminate what isn't wrong?

Just to reiterate, I really don't want to start speculating about possible causes and take a trial-and-error approach, because it's going to be at least several days at a time before I would have conclusive results. I'm looking for solutions to reliably trace the problem to its source.

share|improve this question
    
You could start by uninstalling Kaspersky. No no, I see you said no guesswork and a methodical approach - upvoted for a good question. Have you done anything like running memtest86, Windows Memory Diagnostic ( oca.microsoft.com/en/windiag.asp ) or other hardware stress test? What vendor is the server and have you installed any of their software e.g. Dell OpenManage or HP Insight? –  TessellatingHeckler Mar 14 '11 at 17:14

3 Answers 3

good place to start

http://blogs.technet.com/b/askperf/archive/2007/09/25/troubleshooting-server-hangs-part-one.aspx

share|improve this answer
1  
For the benefit of future, frustrated, impatient readers, would you consider updating this answer to explain exactly what it is that you're linking to and how it answers the question being asked here? –  Aaronaught Mar 14 '11 at 16:12
    
Tony shouldn't have done a link-drop and run, because that doesn't help with Google search results, but that's a solid article. Point 1 from the summary : "Is this a hard or soft hang? If this is a hard hang, then the odds are that there is an underlying hardware issue, so contact your hardware vendor. " I expanded on this point in my answer. –  mfinni Mar 14 '11 at 16:29
1  
Point 5 is very good too - if you can cause a memory dump, you can open a ticket with PSS, send the dump to microsoft, and get a solid answer. –  mfinni Mar 14 '11 at 16:30
    
@mfinni yes, it is a very useful blog, however, having just encountered three separate answers in the last 24 hours with broken links, I cannot in good conscience upvote or accept an answer with no context. Plus I don't really appreciate answers that literally just link to the first Google result for "troubleshoot server hang" - I'm quite capable of doing that myself. –  Aaronaught Mar 14 '11 at 17:12
    
don't use google use bing that could be your problem! –  tony roth Mar 14 '11 at 17:31

With nothing in the logs at all, and no way to reproduce the problem, you've got a lot less to go on, so it will be tougher to be methodical as you are requesting.

If this is hardware from a top-tier vendor, run their diagnostics. IBM, Dell, HP all have diagnostic suites - and free monitoring suites, as well (Director, SIM, and OpenManage, respectively.)

Chronologically, when did this start happening, and did anything change in or near this server before that point? New hardware installed (and/or drivers), update to AV software, new RAM? You said it's a new server - is it new to you, or new to the organization entirely?

Can you P2V it in a sandbox and see if the problem persists?

Is it possibly related to increased load - can you cause it to happen, or take a guess (or show some graphs) to see if more people are using it at the times it happens?

share|improve this answer
    
I've updated the question. (a) no hardware diagnostics, (b) server is brand new, (c) as far as I know it has been happening ever since it was set up, although it's hard to tell due to the intermittency, (d) definitely not due to increased load, and (e) difficult to virtualize, the whole thing was basically a cost-cutting exercise and we have no virtual hosts on site. Really the point of my question was around setting up some sort of continuous monitoring in order to determine what's failing first, processes starting up or shutting down around that time, etc. Not making guesses after the fact. –  Aaronaught Mar 14 '11 at 17:05
    
OK - there aren't too many monitoring solutions that I've seen that will (out-of-the-box) record what you're looking for - processes starting and stopping. You can probably do some sort of SNMP querying of running processes every polling cycle and log that. Do you have a monitoring infrastructure in place now? If not, you'd have to set one up, and if cost is a concern, that might be a problem. But as I said, it looks like hardware. If you bought hardware without vendor support, you're in a bind, because hardware failing is tough to diagnose if you're not willing to swap parts and test. Sorry. –  mfinni Mar 14 '11 at 17:37
    
But if this machine is running SQL and IIS and nothing else, and no one's logging into it, it won't have "processes stopping and starting" except for anything you've scheduled in SQL Agent or Scheduled Tasks. That's probably not it. You've probably got a hardware problem and you're going to have a tough time finding it with the restrictions you've listed. memtest86 is a good start though. –  mfinni Mar 14 '11 at 17:39
    
I'm not sure that's true; process/services start and stop all the time, even on a clean installation. If I look through the event log on a typical machine I can see all sorts of processes stopping and starting - Windows Modules Installer, WinHTTP Web Proxy Auto-Discovery, etc. IIS starts and stops app pools; then there's SQL Agent and MSDTC. Memory test is a good start, yes; unfortunately there's no equivalent tool for RAID controllers or arrays and so on, and even if there were, a green light on the test doesn't necessarily mean that there's no problem in production. –  Aaronaught Mar 14 '11 at 19:10
1  
Intermittent problems are difficult to diagnose, but given the description I've read so far, this is most likely a hardware problem. (I've had DOA RAM, Hard Drives, and Motherboards from top brands - don't rule it out just because it's new). If this were caused by failures in drivers or other software, you'd almost certainly have event log entries and stop errors. Without them, I'd be looking at heat (are all the fans spinning properly) and RAM first. –  Multiverse IT Mar 14 '11 at 23:22

It's pretty paradoxal, you say you have no hardware diagnostics but you want a methodic way to proceed... hardware diagnostics is the methodical way to proceed for hardware faults.

Otherwise if it's a low level software fault there might (should?) be a memory dump somewhere and Microsoft would provide some tool to analyse it, although they don't provide much documentation to understand low level processes so it might be a dead end.

Might, should would... it's been a long time I experimented with such stuff! The problem is usually that you're dealing with closed source so you're virtually on your own!

Maybe support from Microsoft?

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.