Statistics on RAM malfunction

Does anyone know about any statistics or studies about how often computers have malfunctioning RAM?

Update: My computer is fine! I don't have RAM problems, I'm interested in the statistics. I get bug reports for my software for which one cause could be malfunctioning RAM on the user's computer, and I would like to know how likely that is.

Thanks!

Carl

-
Can you give some specifics about the problem that you are blaming on ram failure ? – Dave Cheney Jun 16 '09 at 13:07
A little. We compute checksums from files, and from parts of those files from the hard drive and once they're loaded into RAM. We've noticed some very strange results on some of the users' systems, which could be explained by bugs or by malfunctioning memory. – Carl Seleborg Jun 18 '09 at 10:20

In a population of server class 36 machine, I see a correctable failure detected by the ECC circuitry once every 3 months.

If you suspect memory failure, you should run `memtest86`, which comes included with just about every popular linux distro these days.

-
How do you monitor that ? – Antoine Benkemoun Jun 18 '09 at 10:28
Most LOM systems keep track of it in their logs. – Chris S Jun 24 '10 at 16:54

From Robin Harris' DRAM error rates: Nightmare on DIMM street:

A two-and-a-half year study of DRAM on 10s of thousands Google servers found DIMM error rates are hundreds to thousands of times higher than thought — a mean of 3,751 correctable errors per DIMM per year.

Harris quotes a study performed over 2.5 years on Google's fleet of servers. Note that servers usually use EEC RAM, which performs some error correction. Consumer-level computers usually don't have this.

Lambda Diode's Berke Durak calculates:

First, let's assume you have a system with no error-correction nor parity. The probability that you'll experience a bit error during the time T will be 1-(1-p)^m .

For T=1 hour , p = 1.3e-12 and m = 4*2^30*8 that gives 0.044 or 4.4% . That is quite a high probability. Indeed, in one day, that leads to a probability of 66% and in 72 hours to a probability of 96% .

So the probability of having at least one bit error in 4 gigabytes of memory at sea level on planet Earth in 72 hours is over 95% .

I won't laugh the next time a colleague says "cosmic ray" when we fail to identify the cause of a crash...

-
"20% of the machines with errors make up more than 90% of all observed errors", "the study found that error rates were motherboard dependent". I think I'll stick with conventional wisdom for the time being. The study smells of "lie, damn lies, and statistics". (just my 2 cents) – Chris S Jun 24 '10 at 17:07

You could boot the computer with memtest86+ and run a check overnight. That's how I find problems.

Yes, I have seen sticks of memory go bad where they would only fail with one particular pattern of memory writes. The BIOS of the computer did not detect the problem, but memtest86 found it on an overnight run.

I've seen two sticks of RAM go bad out of about fifty computers that I've used over the past ten years. It happens, but not often.

-
Another vote for memtest86+. It walks your memory bit by bit looking for errors. – Dave Drager Jun 16 '09 at 13:10
Thanks guys, but I really need stats: the problem does not occur on my computer, but on the user's computer (and we have 200000+ users). – Carl Seleborg Jun 16 '09 at 13:23

You might want to have a look at this google study :

On average, about one in three Google servers experienced a correctable memory error each year and one in a hundred an uncorrectable error

-

I've seen a handful of memory modules fail outright in operational servers over the last decade or so and a slightly higher number of failures when doing Memtest86 burn in tests on newly delivered hardware. These are server systems, almost all of which will have ECC memory of one sort or another so I'd expect much more frequent issues on client systems with non-error correcting RAM. I don't have a huge sample set to work from though, we have a couple of dozen servers of our own and in terms of commissioning customer systems I'd say I've worked on a hundred or so at a level where I'd actually be paying attention to the RAM.

On the client side I have a bit more experience at enterprise scale - I was a senior engineer for a group managing 50k end user PC's for a couple of years and we never saw RAM hard or soft failures as a significant problem, certainly it was not something that affected any measurable percentage of systems. That's not to say it didn't happen, just that I'd be very surprised if it was a problem that affected >1% of business class desktops and notebooks. Some specific models would demonstrate really high failure rates that were related to build quality control, the first batch of IBM Thinkpad T30's had an issue with their second DIMM slot that led to us having to repair\replace a couple of thousand machines at one point.

This blog post from Microsoft's Larry Osterman from 2005 might give a possible explanation for some of these though - his analysis of some weird errors reported in the fairly large dataset that comes from Windows Error Reporting indicates that many of those strange problems are caused by over-clocking. If a significant number of your end users are likely to be using over-clocked consumer level kit then this may be related to your errors.

-

If you are running Linux:

If you don't want to reboot into memtest86+ you can get some results by running memtester for testing the memory to find if it is faulty or not. It does a realistically good job for finding the irregular faults as well as with the non-deterministic faults in it. It has several tests for catching the borderline of the memory, and produces a verbose report of faults located, tests run, and the time taken for finding the faults in the computer. No need to reboot you can run it on a running Linux system.

I didn't find any link for the app but here is the debian package information:

-
I'm sorry, but my question was not about my own system. Please read more carefully. – Carl Seleborg Feb 1 '10 at 13:26

Do you have the option to use 'mirrored memory' in your system - that would tell you if you have memory problems or not - with that in place there's MUCH less chance that any errors are due to physical memory problems.

-
Thanks Chopper3, but again: the question was about statistics. My own computer is fine and I can't ask 200000+ users to use mirrored memory :-) – Carl Seleborg Jun 18 '09 at 10:17
Good point, well made - wasn't aware of the scope however. – Chopper3 Jun 18 '09 at 11:01