1

I've been having issues with a hard drive in a dedicated CentOS server with the following symptoms:

  1. Random MySQL Database Corruption -- Tables have been corrupted and repaired multiple times. Several hours after each repair they get corrupted again. MySQL has been completely uninstalled and a new version installed and databases restored from backups but after a few hours tables were corrupt again.
  2. Random Rsync Transfer Corruption -- While transferring a 2GB database backup from another server the file was corrupted several times before finally transferring successfully.
  3. Random File Copy Corruption -- While testing I copied a 6GB database backup to another file (cp file1 file2) and tested the two files using md5sum and received different signatures. This is repeatable: each time I copy the file I get a different md5 signature. Once copied the md5 signature doesn't change.

Tests I have performed:

  1. File Copy Tests -- As mentioned above, copying a large file seems to result in corruption of the new copy and seems to be write based.
  2. SmartCtl -- Checking and testing the drive with smartctl doesn't reveal any issues. I've had other hard drive failures and issues have been obvious with smartctl.
  3. System Logs -- There are no IO error messages in the logs unlike the previous hard drive issues I've had with other servers.

At this point my questions are:

  1. What other tests can I perform to confirm the issue? I only have remote access to the server.
  2. Might there be other potential sources of this issue other than the hard drive?
  3. Is there any way to monitor or check for this issue in the future?

I just want to make sure the issue is the hard drive before I submit a service request to change it (if it is not the hard drive it could end up costing me ).

Update: I had the hard drive changed but had the same corruption error when copying large files so it wasn't the drive unless I'm lucky enough to get two drives with the exact same issue. Using the "cmp" command I found it was always the MSB of the byte that was flipped from 0 to 1 in the corrupted file. I'm having the entire server hardware swapped so I won't know the exact cause of the issue.

2 Answers 2

1

Harddrive errors tend to be caught by the kernel. Does your server have ECC RAM (it should)... without it memory errors can be missed. Same with any cache RAM on RAID adapters and the like. Pull the DIMMs, clean the contacts and try again, or try running Memtest.

Checking for SMART errors on the drives can be helpful. Drives can fail without SMART errors, but usually marginal drives will have them. "smartctl -a /dev/sd[x]" or smartctl --test=long /dev/sd[x] should give some more information.

2
  • I believe its Ram too...
    – Arenstar
    Nov 8, 2010 at 17:09
  • Smartctl shows no errors after a "long" test which is why I'm questioning whether it is really a HD error or not.
    – uesp
    Nov 8, 2010 at 17:29
0

Do you see HD write errors in the kernel log? Make sure to check the S.M.A.R.T. stats of the drive, e.g.: http://www.captain.at/howto-linux-smartmontools-smartctl.php

Also, corrupted memory could cause corrupted files. Do you have remote KVM access to the machine? In this case you should run memtest or similar (http://www.memtest.org/)

1
  • From what I can see there are no IO related errors in the logs (at least in /var/log/messages). I can get a KVM attached and will looked in memtest for RAM testing. Thanks.
    – uesp
    Nov 8, 2010 at 17:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .