ECC CE (Correctable Error) occuring every 5 minutes exactly

Question

On one of our computing nodes I am getting ECC CE (correctable errors). What is a little bit peculiar about is is that errors are not massive, just a single occurrence exactly every 5 minutes.

messages.log:

May  7 11:43:37 armada9 kernel: [22220081.676263] EDAC MC1: 1 CE on unknown memory (csrow:4 channel:1 page:0x41daad offset:0xc30 grain:0 syndrome:0x2254)
May  7 11:48:37 armada9 kernel: [22220381.919057] EDAC MC1: 1 CE on unknown memory (csrow:4 channel:1 page:0x407bb8 offset:0x150 grain:0 syndrome:0x33a8)
May  7 11:53:37 armada9 kernel: [22220682.161798] EDAC MC1: 1 CE on unknown memory (csrow:4 channel:1 page:0x41e6bd offset:0x6a0 grain:0 syndrome:0x33a8)
May  7 11:58:37 armada9 kernel: [22220982.404501] EDAC MC1: 1 CE on unknown memory (csrow:4 channel:1 page:0x427c14 offset:0x880 grain:0 syndrome:0x33a8)
May  7 12:03:37 armada9 kernel: [22221282.647210] EDAC MC1: 1 CE on unknown memory (csrow:4 channel:1 page:0x426e88 offset:0x830 grain:0 syndrome:0x33a8)

syslog example entry:

May  7 12:03:37 armada9 kernel: [22221282.647114] [Hardware Error]: MC4 Error (node 1): DRAM ECC error detected on the NB.
May  7 12:03:37 armada9 kernel: [22221282.647210] EDAC MC1: 1 CE on unknown memory (csrow:4 channel:1 page:0x426e88 offset:0x830 grain:0 syndrome:0x33a8)
May  7 12:03:37 armada9 kernel: [22221282.647215] [Hardware Error]: Error Status: Corrected error, no action required.
May  7 12:03:37 armada9 kernel: [22221282.647299] [Hardware Error]: CPU:6 (10:8:0) MC4_STATUS[Over|CE|MiscV|-|AddrV|CECC]: 0xdc54400033080813
May  7 12:03:37 armada9 kernel: [22221282.647393] [Hardware Error]: MC4_ADDR: 0x0000000426e88830
May  7 12:03:37 armada9 kernel: [22221282.647443] [Hardware Error]: cache level: L3/GEN, mem/io: MEM, mem-tx: RD, part-proc: SRC (no timeout)

Another thing than baffles me is that cat /sys/devices/system/edac/mc/mc*/csrow*/ce_count shows 4x 0. dmidecode -t memory | grep Size reports there are 8x 2GB dice installed. But cat /sys/devices/system/edac/mc/mc*/csrow*/size_mb shows 4x 4096. I am guessing that the memory chips are single ranked, and pairs of dice got coupled. Is this thinking right? Still it does not explain why error count is 0.

This is going on for about 2-3 days already. Every error so far was reported as corrected, but this is pretty annoying and probably not safe.

Is the RAM die dying and I am lucky that it's just some system process happened to be placed in there (as opposed to computation)? I don't think I have anything running every 5 minutes, but maybe some logging tools are.

Or the reason can be something else?

Answer 1

A similar problem happened when I installed new DIMMs in my PowerEdge R815. I thought one of the DIMMs was bad, but didn't know which of the 32 DIMMs it might be. It turned out that the hardware's LCD panel (and the hardware log) reported the failure, and provided the DIMM slot id. When I reseated the DIMM, the error went away -- so it wasn't an error that could be corrected by ECC after all.

Answer 2

It is important to map the csrow and channel to physical slot/DIMM and replace ASAP. In my experience you will start to get more and more errors but it all depends on how fast the chip goes totally bad, I have seen it progress from a few errors a day to dead the next day or it could make it for several months or more (all depending on your workload). Eventually it will be to the point that your console is filled with them and eventually it leads to a UE (Uncorrectable Error) and your server will crash and DIMM will be unusable.

One other important thing is that if your BIOS (and most server BIOS's will do this) detects multiple bit failures it may disable that DIMM slot. Do not wipe/erase or reset the enabled DIMMs in the BIOS otherwise your server will probably not boot at all (as in no POST) and unless you recall which DIMM was marked bad you will have to remove chip after chip until it boots to figure out which one was bad (big pain in a corporate or datacenter setting). Also if replacing a DIMM that has been marked bad you will probably have to re-enable it or wipe the record of the bad DIMM from your BIOS for it to be recognized.