I've been having issues with a hard drive in a dedicated CentOS server with the following symptoms:
- Random MySQL Database Corruption -- Tables have been corrupted and repaired multiple times. Several hours after each repair they get corrupted again. MySQL has been completely uninstalled and a new version installed and databases restored from backups but after a few hours tables were corrupt again.
- Random Rsync Transfer Corruption -- While transferring a 2GB database backup from another server the file was corrupted several times before finally transferring successfully.
- Random File Copy Corruption -- While testing I copied a 6GB database backup to another file (cp file1 file2) and tested the two files using md5sum and received different signatures. This is repeatable: each time I copy the file I get a different md5 signature. Once copied the md5 signature doesn't change.
Tests I have performed:
- File Copy Tests -- As mentioned above, copying a large file seems to result in corruption of the new copy and seems to be write based.
- SmartCtl -- Checking and testing the drive with smartctl doesn't reveal any issues. I've had other hard drive failures and issues have been obvious with smartctl.
- System Logs -- There are no IO error messages in the logs unlike the previous hard drive issues I've had with other servers.
At this point my questions are:
- What other tests can I perform to confirm the issue? I only have remote access to the server.
- Might there be other potential sources of this issue other than the hard drive?
- Is there any way to monitor or check for this issue in the future?
I just want to make sure the issue is the hard drive before I submit a service request to change it (if it is not the hard drive it could end up costing me ).
Update: I had the hard drive changed but had the same corruption error when copying large files so it wasn't the drive unless I'm lucky enough to get two drives with the exact same issue. Using the "cmp" command I found it was always the MSB of the byte that was flipped from 0 to 1 in the corrupted file. I'm having the entire server hardware swapped so I won't know the exact cause of the issue.