Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Our new server has been running basically fine for a few months. Twice, however, it shut itself down for no apparent reason.

The most recent occurrence was at 11:41pm a few days ago. The event logs show nothing untoward, and the last entry is a fairly mundane audit entry in the Security log. The UPS log shows no power issues. Nothing in particular was running, as it was after hours. Except of course the nightly backup, which starts at 10pm. The backup log also shows nothing interesting and just stops in the middle of the backup. Although the server is configured to write a kernel dump and restart, there is no memory dump and the system did not restart. It's an HP Proliant ML330 G6 Series server.

When the server was restarted manually the following morning, the following events were logged:

Log Name:      System
Source:        EventLog
Date:          4/16/2011 8:20:22 AM
Event ID:      6008
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      XXXXXXXX.xxxxxxxxxxxxxxxx.local
Description:
The previous system shutdown at 11:41:26 PM on ‎4/‎15/‎2011 was unexpected.

and

Log Name:      System
Source:        Microsoft-Windows-Kernel-Power
Date:          4/16/2011 8:20:00 AM
Event ID:      41
Task Category: (63)
Level:         Critical
Keywords:      (2)
User:          SYSTEM
Computer:      XXXXXXXX.xxxxxxxxxxxxxxxx.local
Description:
The system has rebooted without cleanly shutting down first. This error could be
caused if the system stopped responding, crashed, or lost power unexpectedly.

and

Log Name:      System
Source:        USER32
Date:          4/16/2011 8:22:34 AM
Event ID:      1076
Task Category: None
Level:         Warning
Keywords:      Classic
User:          XXXXXXXXXXXXXXX\Administrator
Computer:      XXXXXXXX.xxxxxxxxxxxxxxxx.local
Description:
The reason supplied by user XXXXXXXXXXXXXXX\Administrator for the last unexpected 
shutdown of this computer is: Other Failure: System Unresponsive
Reason Code: 0x8000005
Problem ID: 
Bugcheck String: 
Comment: 

I've spent some time researching this and found very little of use. Anyone have any ideas?

UPDATE: Here are the relevant portions of the iLO2 log:

305 04/15/2011 23:42:00 Server reset. 
306 04/15/2011 23:42:00 Server power removed. 
307 04/15/2011 23:42:00 iLO 2 network link down. 
308 04/15/2011 23:42:00 iLO 2 network link up at 100 Mbps. 
309 04/16/2011 08:17:00 Server power restored. 

UPDATE: I increased the size of the paging file to allow for full kernel dumps, so if it's really a Windows crash, I'll be able to see what happened - the next time it happens.

UPDATE: The server firmware was already up to date.

UPDATE: There were a lot of updates available for drivers and system software. I've installed most of them and now I'm just waiting to see if the problem happens again.

share|improve this question
    
dang do you have ASR enabled? –  tony roth Apr 18 '11 at 20:43
    
@tony: Do you you mean the Automated System Recovery built into Windows or the HP Automatic Server Recovery? –  boot13 Apr 18 '11 at 21:51
    
the intergrated log management should show the ASR, post that log if you can. If it doesn't show anything I still suspect the ASR process. do you see event id 57 in your system log? –  tony roth Apr 19 '11 at 2:11
    
@tony: Sorry, still not following. By ASR do you mean the HP 'Automatic Server Recovery'? If so, where can I find it? When you refer to the 'integrated log management' do you mean the 'Integrated Management Log' in the System Management Homepage? I can confirm that there are no events with ID 57 in the Windows System event log. –  boot13 Apr 19 '11 at 18:10
    
Sorry I wasted your time but I think you are correct "23:42:00 Server power removed" I think means that an external source distrubted the power! –  tony roth Apr 19 '11 at 20:27
show 9 more comments

6 Answers

It's most likely a faulty power switch/LED cable kit. My ML310 G5 was doing the same thing, and that is what fixed the problem. Apparently, it is a known issue with HP.

459186-001-02 HEWLETT-PACKARD PROLIANT ML310 G5 SYSTEM FRONT LED TO SYS/BRD CABLE P/N: 459186-001-02 - HEWLETT-PACKARD ORIGINALS

share|improve this answer
    
Thanks for the info. I'll definitely look into that. –  boot13 Oct 21 '12 at 13:27
add comment

Is the machine overheating? Check the fans and vents for dust bunnies.

share|improve this answer
    
@ed: Nope. Brand new server, fans all running fine, temps all nominal. –  boot13 Apr 18 '11 at 21:52
1  
Hmm. Perhaps run a stress test tool, like Intel Burn Test to make sure the server isn't faulty first. Google Intel Burn Test. –  ed209 Apr 19 '11 at 13:52
    
@ed: Okay, thanks for the tip. I'll run that as soon as I can find a window of opportunity and post the results. –  boot13 Apr 19 '11 at 18:14
add comment

Do you have the HP management agent software installed? You mention Windows event logs and backup logs but not the "hardware" logs. You need to look there too because spontaneous shutdowns might be related to a hardware issue that you won't be able to see info about anywhere else.

share|improve this answer
    
@icky: Okay, I looked at the System Management Home Page and the logs therein: System Management Homepage Log, Integrated Management Log, HP Version Control Agent Log and Integrated Lights-Out 2 Log. The only one that shows anything interesting is the iLO2 log, excerpts of which I will add to my question. –  boot13 Apr 18 '11 at 23:58
    
My theory was/is that the absence of a kernel dump means that it wasn't Windows that crashed so it must be hardware or power related. It's unfortunate there's nothing useful in the HP logs. How about tracing the power back - dual power supplies? cord in a place where it could snag when someone walks by? connected to a PDU that other servers are on or ? on the same circuit as other things? That kind of stuff. –  icky3000 Apr 19 '11 at 3:23
    
@icky: I think you're probably right that Windows wasn't the culprit. Still, I will increase the size of the paging file as I understand that at its current size it may not allow for full kernel dumps. As for the power side, there's only one power supply, so that's a possibility; no cord issues as the server is in a (well-ventilated) closet and things are tucked away appropriately; and the server is on a solid, new UPS that is functioning perfectly and shows nothing untoward in its history. –  boot13 Apr 19 '11 at 18:25
    
There's probably a web interface for the UPS too and you should look into that and its logs. There might be something in there like it purposely shut down that power connection for some reason. –  icky3000 Apr 19 '11 at 19:28
    
@icky: Yeah, I checked that. It has logs and some nice graphs of the power coming into and leaving the UPS and throughout the period in question everything was smooth. Kind of disappointing, really: the previous time this occurred we didn't yet have the UPS so I pretty much assumed it was a power glitch. It could still be the server's power supply, though. –  boot13 Apr 19 '11 at 21:10
show 1 more comment

I had this EXACT issue happening on my Server 2008 R2 box. It turns out that the Xeon 5000 series CPUS, which your machine does use, have an issue with 2008 R2 and Hyper-V role. I'm going out on a limb here and assuming you have the Hyper-V role installed, based on the issue being identical to the one I was having.

There is a hotfix from Microsoft available HERE. I installed it on my system, and it has been trouble free since then.

share|improve this answer
    
Dang. I thought you were onto something there, but we don't have that role installed. Can you describe the process you went through to determine that the Hyper-V role was the problem? It might help me to know where you looked. –  boot13 May 3 '11 at 20:49
    
The link I provided was the answer for me. I was researching the issue, and that article came up. However, it still may be worthwhile to install the hotfix, even though it mentions the Hyper-V role and you dont have it installed. –  DanBig May 3 '11 at 20:50
    
Okay, thanks. I'll keep that one in mind in case the firmware and driver updates don't fix it. –  boot13 May 3 '11 at 22:24
add comment

I'm going to go waaaaaaay out on a limb here, and say that you might need a firmware update. Source. We had something similar with our DL380 G6 a while back.

share|improve this answer
    
That looks promising. I'll be installing that firmware update shortly. Thanks! –  boot13 May 3 '11 at 22:26
    
Dang. Finally scheduled a time to do this, and it turns out we're already running the latest firmware, including the SPLD. Back to the other suggestions... –  boot13 May 12 '11 at 16:23
add comment

If that really was a system crash, you would have found an event such as this in the System log:

Level: Error
Source: Bugcheck
Event ID: 1001
Text: The computer has rebooted from a bugcheck.  The bugcheck was: [...]

Also, being configured to save a kernel dump and then reboot, the server would have done just that.

The absence of such an event log and of a subsequent reboot means the shutdown was caused by an external event (power missing, hardware fault...). Also, your ILO logs seem to confirm that a power failure was the actual reason.

share|improve this answer
    
Power does seem to have been the issue, but since the UPS didn't report anything unusual, I'm going to assume it was a faulty power switch assembly as suggested by Cole. Thanks! –  boot13 Oct 21 '12 at 13:30
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.