Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I've been having issues with a hard drive in a dedicated CentOS server with the following symptoms:

  1. Random MySQL Database Corruption -- Tables have been corrupted and repaired multiple times. Several hours after each repair they get corrupted again. MySQL has been completely uninstalled and a new version installed and databases restored from backups but after a few hours tables were corrupt again.
  2. Random Rsync Transfer Corruption -- While transferring a 2GB database backup from another server the file was corrupted several times before finally transferring successfully.
  3. Random File Copy Corruption -- While testing I copied a 6GB database backup to another file (cp file1 file2) and tested the two files using md5sum and received different signatures. This is repeatable: each time I copy the file I get a different md5 signature. Once copied the md5 signature doesn't change.

Tests I have performed:

  1. File Copy Tests -- As mentioned above, copying a large file seems to result in corruption of the new copy and seems to be write based.
  2. SmartCtl -- Checking and testing the drive with smartctl doesn't reveal any issues. I've had other hard drive failures and issues have been obvious with smartctl.
  3. System Logs -- There are no IO error messages in the logs unlike the previous hard drive issues I've had with other servers.

At this point my questions are:

  1. What other tests can I perform to confirm the issue? I only have remote access to the server.
  2. Might there be other potential sources of this issue other than the hard drive?
  3. Is there any way to monitor or check for this issue in the future?

I just want to make sure the issue is the hard drive before I submit a service request to change it (if it is not the hard drive it could end up costing me ).

Update: I had the hard drive changed but had the same corruption error when copying large files so it wasn't the drive unless I'm lucky enough to get two drives with the exact same issue. Using the "cmp" command I found it was always the MSB of the byte that was flipped from 0 to 1 in the corrupted file. I'm having the entire server hardware swapped so I won't know the exact cause of the issue.

share|improve this question

2 Answers 2

up vote 1 down vote accepted

Harddrive errors tend to be caught by the kernel. Does your server have ECC RAM (it should)... without it memory errors can be missed. Same with any cache RAM on RAID adapters and the like. Pull the DIMMs, clean the contacts and try again, or try running Memtest.

Checking for SMART errors on the drives can be helpful. Drives can fail without SMART errors, but usually marginal drives will have them. "smartctl -a /dev/sd[x]" or smartctl --test=long /dev/sd[x] should give some more information.

share|improve this answer
    
I believe its Ram too... –  Arenstar Nov 8 '10 at 17:09
    
Smartctl shows no errors after a "long" test which is why I'm questioning whether it is really a HD error or not. –  uesp Nov 8 '10 at 17:29

Do you see HD write errors in the kernel log? Make sure to check the S.M.A.R.T. stats of the drive, e.g.: http://www.captain.at/howto-linux-smartmontools-smartctl.php

Also, corrupted memory could cause corrupted files. Do you have remote KVM access to the machine? In this case you should run memtest or similar (http://www.memtest.org/)

share|improve this answer
    
From what I can see there are no IO related errors in the logs (at least in /var/log/messages). I can get a KVM attached and will looked in memtest for RAM testing. Thanks. –  uesp Nov 8 '10 at 17:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.