Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

About a week ago I experienced one very interesting situation. I had a workstation - old desktop with Asus P5LD2 motherboard and 4 x 1 GB non-registered DDR2 Kingston memory. That same machine was a victim of a power stroke quite some time ago, IIRC 12-14 months ago. At the time of the power stroke the PSU fried and the HDD died. I replaced both, ran tests, including memtest and everything seemed fine. The user was working happily on it, until one day last week when he found some recent data "corruption" in some of his files. I investigated the issue and managed to narrow it down to motherboard fault. However, the "data corruption" was rather interesting and reproducible:

  • copying text files from local directory to another local directory and running diff between both versions, there was only 1 bit changed somewhere random in the file;
  • this bit was always the 6th out of 8, viewed in hex text editor, i.e. hex 19 becomes hex 39;
  • the issue was reproducible while accessing NFS mounts and local mounts. Same exact tests repeated from other clients produced no differences;
  • while copying from this machine over the network with rsync -av the command failed with Corrupted MAC on input. Disconnecting: Packet corrupt;
  • tried same MB, but different memory set - again differences;
  • old memory set on another Asus P5LD2 MB - no differences;
  • memtest ran for more than 24 hours - no single error reported.

Conclusion from the tests - the bit flipping occurs only on this exact machine, regardless of the memory set used and the data location (local or NFS).

Based on all my tests, the only components left in the equation are the motherboard and the CPU.

My question(s) are:

  1. what causes the bit flipping and how exactly it happens?;
  2. is there a way to detect it?;
  3. how to test/probe for it, when memtest fails?

I still have the troublesome machine in-house and am willing to run any tests to learn more about this.

The OS is Ubuntu Lucid 10.04, 64-bit.

Edit I forgot to mention that most (if not all) capacitors on the MB where bended on top, instead of flat.

share|improve this question
    
+1 good research –  Bigbio2002 Jan 17 '13 at 22:40

1 Answer 1

up vote 1 down vote accepted

Sounds like a problem with the CPU accessing peripherals like the disk controller and network card. It could be the northbridge overheating. When the CPU is hot, the northbridge gets hotter than otherwise. It could also be the CPU overheating.

During memtest, there's minimal I/O and minimal CPU work.

I forgot to mention that most (if not all) capacitors on the MB where bended on top, instead of flat.

That will cause the DC power supplied to components like the RAM, CPU, and northbridge to get noisy as load goes up. That could easily be the cause of your problem. I'd say the motherboard should be retired.

share|improve this answer
    
+1 for the overheating suggestion. I posted an update in my question, regarding the capacitors condition, which may be related. –  grs Jan 17 '13 at 23:20
    
If the capacitors are bulging, the motherboard should just be retired. They're not doing their job properly anymore. –  David Schwartz Jan 17 '13 at 23:25
    
Any idea how I can detect bit-flipping proactively, especially when the RAM is desktop-class one? –  grs Jan 17 '13 at 23:44
1  
You have. You've now detected that there will be bit flips. :) –  David Schwartz Jan 18 '13 at 0:03

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.