1

Disclaimer: I'm running an aging cluster that I have very limited information for - I'm about the tenth graduate student to handle day to day maintenance, and I am no expert on HPC, hardware, or software. I use this cluster to run the code I have developed, and troubleshoot problems only when necessary. Documentation is limited to mainly the cluster specific propriety cluster management system (Scyld), and we are no longer under warranty, so calling tech support is a last resort, as we would be charged (academia strongly advises against this).

Problem: One of the nodes for this cluster will not boot, giving the following error during boot:

Node1: Dram ECC requested but not enabled
Node1: No DQS Receiver Enable pass window found
Node1: DQS Rcvr En pass window too small (far right of dynamic range)
DQS-DQ passing window too small
Press F1 to Resume

Upon hitting F1, the node reboots and comes back to this. This makes me think that there is a memory (read: hardware) error. I opened the BIOS, and all the memory shows up - but there are 5 items in the event log (posted at the bottom). I have a feeling that these codes can direct me to the error - specifically if I have a dead stick of DIMM, or something worse.

Attempted Fixes: Each node on the cluster contains 2x Quad-Core AMD Opteron Processor 2350, and each processor have their own, 8 slot memory banks. Currently, each bank has 4 sticks of DIMM installed in slots 1-4. I've attempted to reseat the DIMM, reverse the order of the DIMM, put the DIMM in slots 5-8, and switch the banks in which the DIMM reside to see if that could allow me to isolate any problems - to no avail.

I've found some information on SEL error codes (here), and this seems somewhat hopeful - but the codes aren't exactly the same, and thus I cannot fully trust them.

Desired Resolution: Figure out what these SEL codes mean, pinpoint the problem, (and hopefully) fix it, and boot the node. Even narrowing it down to a bad stick of DIMM, bad CPU, or anything would be helpful.

Thanks!

Other information that may or may not be helpful: The system is a Penguin cluster with 32 nodes, with each node identical (described in detail above). It's running CentOS 4.6. It's a beowulf cluster used for highly parallel computational simulations, using MPI to gather/scatter processes to nodes.

System Event Log:

SEL Entry Number            [1]
SEL Record ID:              0001
SEL Record Type:                02 (System Event)
Event Timestamp:                Sep 21, 2016 14:01:52
Generator ID:               0001
Event Message Format Ver:       04 (IPMI ver 1.5)
Event Sensor Type:          C1
Event Sensor Number:            81
Event Dir Type:             70
Event Data:                 A0 01 07

SEL Entry Number            [2]
SEL Record ID:              0002
SEL Record Type:                02 (System Event)
Event Timestamp:                Sep 21, 2016 14:02:03
Generator ID:               0001
Event Message Format Ver:       04 (IPMI ver 1.5)
Event Sensor Type:          0F (POST Error)
Event Sensor Number:            82
Event Dir Type:             70
Event Data:                 A0 08 41

SEL Entry Number            [3]
SEL Record ID:              0003
SEL Record Type:                02 (System Event)
Event Timestamp:                Sep 21, 2016 14:02:03
Generator ID:               0001
Event Message Format Ver:       04 (IPMI ver 1.5)
Event Sensor Type:          0F (POST Error)
Event Sensor Number:            82
Event Dir Type:             70
Event Data:                 A0 0B 41

SEL Entry Number            [4]
SEL Record ID:              0004
SEL Record Type:                02 (System Event)
Event Timestamp:                Sep 21, 2016 14:02:04
Generator ID:               0001
Event Message Format Ver:       04 (IPMI ver 1.5)
Event Sensor Type:          0F (POST Error)
Event Sensor Number:            82
Event Dir Type:             70
Event Data:                 A0 0D 41

SEL Entry Number            [5]
SEL Record ID:              0005
SEL Record Type:                02 (System Event)
Event Timestamp:                Sep 21, 2016 14:02:04
Generator ID:               0001
Event Message Format Ver:       04 (IPMI ver 1.5)
Event Sensor Type:          0F (POST Error)
Event Sensor Number:            82
Event Dir Type:             70
Event Data:                 A0 0F 41
memory event-log cluster

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.