2

I'm running ubuntu server 14.04 on Supermicro X10SLM-F / Xeon E3-1271 v3

Memory: SuperTalent 32GB DDR3 1600 ECC

About every 4 days, the logs on Ubuntu will show this:

{1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 1
{1}[Hardware Error]: It has been corrected by h/w and requires no further action
{1}[Hardware Error]: event severity: corrected
{1}[Hardware Error]:  Error 0, type: corrected
{1}[Hardware Error]:  fru_text: CorrectedErr
{1}[Hardware Error]:   section_type: memory error
[Firmware Warn]: error section length is too small

Immediately after this the server reboots itself in a "power-cycle" fashion.

When I look in the BIOS event log, I see this:

DATE            TIME           ERROR CODE      SEVERITY
06/13/15      13:13:38      Smbios 0x02         P1-DIMMB2

And the description of the error is:

Single Bit ECC Memory Error

ipmitool in Ubuntu show this:

ipmitool sel elist
...
...
  1a | 06/13/2015 | 13:13:39 | Memory | Correctable ECC | Asserted | CPU 0 DIMM 8
  1b | 06/13/2015 | 13:13:39 | Memory | Uncorrectable ECC | Asserted | CPU 0 DIMM 8

A few questions:

  1. If the ECC memory is self correcting, why does the machine reboot itself?

  2. Am I, perhaps, missing some setting in the BIOS that will stop the box from rebooting itself?

  3. Is this obviously a memory stick issue or can this be a slot issue or a CPU issue?

  4. How to stop the server from rebooting?

Thank you for any advice.

  • Probably there were several bit errors. It just happened that the first one which got detected was correctable, and the next which got detected was probably an uncorrectable multi-bit error. – kasperd Jun 16 '15 at 21:19
  • My Supermicro BIOS has an option named "Single bit ECC assertion". If enabled, the hardware reboots after 1-bit errors are spotted and properly corrected. This option should be toggled to "Double bit ECC assertion" to let ECC correct memory errors. Single bit flips are normal, up to some recommended threshold, which IIRC is 10 per hour nowadays. – davide Dec 14 '16 at 14:22
1

The system should not reboot upon correctable memory error. Do you see additional information/pattern via ipmitool sel elist ? The BMC watchdog could reboot the system, check if it is enabled via ipmitool mc watchdog get. As you already have the information on the location of the bad memory module, replace it and if the problem manifests again, the memory slot could be at fault.

X10SLM-F the RAM that you use is not on the list of tested RAM modules - if you have the possibility, replace all the memory bars in a 'problem' system with equivalent Supermicro-tested ones. Also, check the list of supported OS for you Ubuntu version.

Related to the CMOS settings, you could use Supermicro SUM, provided you have the SUM keys installed, to dump the BIOS settings from all the systems then vimdiff them to see if there is any CMOS parameter being different for the systems that regularly reboot compared to the system(s) that do not.

sum -i <IP Address of the BMC> -u <BMC user> -p <BMC password> -c GetCurrentBiosCfg --file myconf.conf
  • ipmitool sel elist command give this: <br/> Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Get SEL Info command failed – Kevin Kelly Jun 15 '15 at 23:13
  • Sorry, had to load modules for ipmitool to work. I updated the original post with the information from ipmitool sel elist – Kevin Kelly Jun 15 '15 at 23:47
  • @KevinKelly you also have uncorrectable ECC proceed to changing the DIMM. – 7y7 Jun 16 '15 at 6:41
  • Well but here is the caveat. I have 5 of those Supermicro servers and all are using the same type of memory. This has happened to 3 of them. Only one has it happening every 4 days, but this same error has caused reboots on other machines. This makes me feel like perhaps the memory is just part of the issue and there is, in fact, something else I am missing in the bios config. – Kevin Kelly Jun 16 '15 at 9:50
0

It's Supermicro hardware, so it's inexpensive and lacking the polish and integration of a Dell, HP or IBM...

ECC RAM is error-correcting, but there's a threshold that's likely being exceeded. The DIMM in question is likely failing and you should plan to replace it.

You can attempt to identify the module by slot and replace it. Since this occurs with frequency it'll be easy to determine the problem.

Also, look at the Related Questions to the right side of this question.

0

I have seen the same problem with this board, rev 1.02. I think it has a specific problem. I buy a lot of SM boards and they are usually pretty good. I think that there is a problem with this board if all DIMMS are populated. I am using windows and I get a blue screen, for example.

Try just running with 16GB (only 2 slots) and I bet you the problem will go away. I know this is not a solution, but would help diagnose the strange behavior. I even sent a board back to SM and they said it was fine, maybe they did not test with 4 DIMMs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.