6

Our new server has been running basically fine for a few months. Twice, however, it shut itself down for no apparent reason.

The most recent occurrence was at 11:41pm a few days ago. The event logs show nothing untoward, and the last entry is a fairly mundane audit entry in the Security log. The UPS log shows no power issues. Nothing in particular was running, as it was after hours. Except of course the nightly backup, which starts at 10pm. The backup log also shows nothing interesting and just stops in the middle of the backup. Although the server is configured to write a kernel dump and restart, there is no memory dump and the system did not restart. It's an HP Proliant ML330 G6 Series server.

When the server was restarted manually the following morning, the following events were logged:

Log Name:      System
Source:        EventLog
Date:          4/16/2011 8:20:22 AM
Event ID:      6008
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      XXXXXXXX.xxxxxxxxxxxxxxxx.local
Description:
The previous system shutdown at 11:41:26 PM on ‎4/‎15/‎2011 was unexpected.

and

Log Name:      System
Source:        Microsoft-Windows-Kernel-Power
Date:          4/16/2011 8:20:00 AM
Event ID:      41
Task Category: (63)
Level:         Critical
Keywords:      (2)
User:          SYSTEM
Computer:      XXXXXXXX.xxxxxxxxxxxxxxxx.local
Description:
The system has rebooted without cleanly shutting down first. This error could be
caused if the system stopped responding, crashed, or lost power unexpectedly.

and

Log Name:      System
Source:        USER32
Date:          4/16/2011 8:22:34 AM
Event ID:      1076
Task Category: None
Level:         Warning
Keywords:      Classic
User:          XXXXXXXXXXXXXXX\Administrator
Computer:      XXXXXXXX.xxxxxxxxxxxxxxxx.local
Description:
The reason supplied by user XXXXXXXXXXXXXXX\Administrator for the last unexpected 
shutdown of this computer is: Other Failure: System Unresponsive
Reason Code: 0x8000005
Problem ID: 
Bugcheck String: 
Comment: 

I've spent some time researching this and found very little of use. Anyone have any ideas?

UPDATE: Here are the relevant portions of the iLO2 log:

305 04/15/2011 23:42:00 Server reset. 
306 04/15/2011 23:42:00 Server power removed. 
307 04/15/2011 23:42:00 iLO 2 network link down. 
308 04/15/2011 23:42:00 iLO 2 network link up at 100 Mbps. 
309 04/16/2011 08:17:00 Server power restored. 

UPDATE: I increased the size of the paging file to allow for full kernel dumps, so if it's really a Windows crash, I'll be able to see what happened - the next time it happens.

UPDATE: The server firmware was already up to date.

UPDATE: There were a lot of updates available for drivers and system software. I've installed most of them and now I'm just waiting to see if the problem happens again.

UPDATE 2018Jun06: after six years of trouble-free operation, this problem has returned, occurring twice in the last week or so. I'm looking into the possibility that the front panel and its wiring are faulty.

UPDATE 2018Nov30: Finally swapped out the front panel cable assembly, but the problem still occurs. Next up is the power supply.

16
  • dang do you have ASR enabled?
    – tony roth
    Apr 18, 2011 at 20:43
  • @tony: Do you you mean the Automated System Recovery built into Windows or the HP Automatic Server Recovery?
    – boot13
    Apr 18, 2011 at 21:51
  • the intergrated log management should show the ASR, post that log if you can. If it doesn't show anything I still suspect the ASR process. do you see event id 57 in your system log?
    – tony roth
    Apr 19, 2011 at 2:11
  • @tony: Sorry, still not following. By ASR do you mean the HP 'Automatic Server Recovery'? If so, where can I find it? When you refer to the 'integrated log management' do you mean the 'Integrated Management Log' in the System Management Homepage? I can confirm that there are no events with ID 57 in the Windows System event log.
    – boot13
    Apr 19, 2011 at 18:10
  • Sorry I wasted your time but I think you are correct "23:42:00 Server power removed" I think means that an external source distrubted the power!
    – tony roth
    Apr 19, 2011 at 20:27

6 Answers 6

5

It's most likely a faulty power switch/LED cable kit. My ML310 G5 was doing the same thing, and that is what fixed the problem. Apparently, it is a known issue with HP.

459186-001-02 HEWLETT-PACKARD PROLIANT ML310 G5 SYSTEM FRONT LED TO SYS/BRD CABLE P/N: 459186-001-02 - HEWLETT-PACKARD ORIGINALS

1
  • I realize that your answer is from six years ago, but I was wondering if you can recall where you heard that this was a known issue with HP. After a long hiatus, this problem has returned, and I'm looking at the front panel/cables as the possible cause.
    – boot13
    Jun 6, 2018 at 12:15
2

I had this EXACT issue happening on my Server 2008 R2 box. It turns out that the Xeon 5000 series CPUS, which your machine does use, have an issue with 2008 R2 and Hyper-V role. I'm going out on a limb here and assuming you have the Hyper-V role installed, based on the issue being identical to the one I was having.

There is a hotfix from Microsoft available HERE. I installed it on my system, and it has been trouble free since then.

3
  • Dang. I thought you were onto something there, but we don't have that role installed. Can you describe the process you went through to determine that the Hyper-V role was the problem? It might help me to know where you looked.
    – boot13
    May 3, 2011 at 20:49
  • The link I provided was the answer for me. I was researching the issue, and that article came up. However, it still may be worthwhile to install the hotfix, even though it mentions the Hyper-V role and you dont have it installed.
    – DanBig
    May 3, 2011 at 20:50
  • Okay, thanks. I'll keep that one in mind in case the firmware and driver updates don't fix it.
    – boot13
    May 3, 2011 at 22:24
2

I'm going to go waaaaaaay out on a limb here, and say that you might need a firmware update. Source. We had something similar with our DL380 G6 a while back.

2
  • That looks promising. I'll be installing that firmware update shortly. Thanks!
    – boot13
    May 3, 2011 at 22:26
  • 1
    Dang. Finally scheduled a time to do this, and it turns out we're already running the latest firmware, including the SPLD. Back to the other suggestions...
    – boot13
    May 12, 2011 at 16:23
1

Is the machine overheating? Check the fans and vents for dust bunnies.

3
  • @ed: Nope. Brand new server, fans all running fine, temps all nominal.
    – boot13
    Apr 18, 2011 at 21:52
  • 1
    Hmm. Perhaps run a stress test tool, like Intel Burn Test to make sure the server isn't faulty first. Google Intel Burn Test.
    – ed209
    Apr 19, 2011 at 13:52
  • @ed: Okay, thanks for the tip. I'll run that as soon as I can find a window of opportunity and post the results.
    – boot13
    Apr 19, 2011 at 18:14
1

Do you have the HP management agent software installed? You mention Windows event logs and backup logs but not the "hardware" logs. You need to look there too because spontaneous shutdowns might be related to a hardware issue that you won't be able to see info about anywhere else.

6
  • @icky: Okay, I looked at the System Management Home Page and the logs therein: System Management Homepage Log, Integrated Management Log, HP Version Control Agent Log and Integrated Lights-Out 2 Log. The only one that shows anything interesting is the iLO2 log, excerpts of which I will add to my question.
    – boot13
    Apr 18, 2011 at 23:58
  • My theory was/is that the absence of a kernel dump means that it wasn't Windows that crashed so it must be hardware or power related. It's unfortunate there's nothing useful in the HP logs. How about tracing the power back - dual power supplies? cord in a place where it could snag when someone walks by? connected to a PDU that other servers are on or ? on the same circuit as other things? That kind of stuff.
    – icky3000
    Apr 19, 2011 at 3:23
  • @icky: I think you're probably right that Windows wasn't the culprit. Still, I will increase the size of the paging file as I understand that at its current size it may not allow for full kernel dumps. As for the power side, there's only one power supply, so that's a possibility; no cord issues as the server is in a (well-ventilated) closet and things are tucked away appropriately; and the server is on a solid, new UPS that is functioning perfectly and shows nothing untoward in its history.
    – boot13
    Apr 19, 2011 at 18:25
  • There's probably a web interface for the UPS too and you should look into that and its logs. There might be something in there like it purposely shut down that power connection for some reason.
    – icky3000
    Apr 19, 2011 at 19:28
  • @icky: Yeah, I checked that. It has logs and some nice graphs of the power coming into and leaving the UPS and throughout the period in question everything was smooth. Kind of disappointing, really: the previous time this occurred we didn't yet have the UPS so I pretty much assumed it was a power glitch. It could still be the server's power supply, though.
    – boot13
    Apr 19, 2011 at 21:10
0

If that really was a system crash, you would have found an event such as this in the System log:

Level: Error
Source: Bugcheck
Event ID: 1001
Text: The computer has rebooted from a bugcheck.  The bugcheck was: [...]

Also, being configured to save a kernel dump and then reboot, the server would have done just that.

The absence of such an event log and of a subsequent reboot means the shutdown was caused by an external event (power missing, hardware fault...). Also, your ILO logs seem to confirm that a power failure was the actual reason.

1
  • Power does seem to have been the issue, but since the UPS didn't report anything unusual, I'm going to assume it was a faulty power switch assembly as suggested by Cole. Thanks!
    – boot13
    Oct 21, 2012 at 13:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .