I'm not a professional sysadmin but since I couldn't find answers to my specific case after researching for a while I was hoping I could get some help here. Our server uses P222 - a HP Smart controller array in RAID1 configuration. I believe some sectors on one of the physical hard drives have failed. I used the
hpacucli tool and the output looked like:-
$ hpacucli ctrl all show config Smart Array P222 in Slot 1 (sn: PDSXH0ARH5I0SW) array A (SATA, Unused Space: 0 MB) logicaldrive 1 (2.7 TB, RAID 1, Ready for Rebuild) physicaldrive 2I:1:1 (port 2I:box 1:bay 1, SATA, 3 TB, OK) physicaldrive 2I:1:2 (port 2I:box 1:bay 2, SATA, 3 TB, Predictive Failure)
I ran the same tool again a few times to check the status and at one point I noticed that "Predictive Failure" was replaced by "Rebuilding 1%" , increased later to 2%. I don't think I did anything to initiate the rebuilding. Anyway, I let it run and checked the status after a while at which point it was back to 'Predictive Failure'.
On running smartctl long and short tests - the selftest logs reported :-
=== START OF READ SMART DATA SECTION === SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: read failure 90% 14368 334201968 # 2 Short offline Completed: read failure 90% 14367 625082211
We are running a MySQL instance on this server and it keeps failing to start complaining of a read error which indicated that it may be due to a failing hard disk/a bad sector hence the tools used above. I had a couple of questions :-
- I don't know for sure but it looks like one of the hard disks is failing partly. In that case, shouldn't the OS (Ubuntu 12.04) just read the data from the mirrored hard disk ? (which would mean MySQL should continue running)
- I was following the steps in http://sg.danny.cz/scsi/badblockhowto.html . The LBA 334201968 (LBA of the long test read failure) corresponds to the data file of MySQL. But I did not want to overwrite any part of this file as I'm not sure if MySQL will permanently see the file as corrupted. What would be my best option to 'fix' the corrupted parts of the disk ?
Happy to report any additional details that may be required for diagnosing/fixing this
EDIT 1: As requested the MySQL error logs like so:-
150824 10:27:00 InnoDB: Completed initialization of buffer pool 150824 10:27:00 InnoDB: highest supported file format is Barracuda. InnoDB: The log sequence number in ibdata files does not match InnoDB: the log sequence number in the ib_logfiles! 150824 10:27:00 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... 150824 10:27:00 InnoDB: Waiting for the background threads to start 150824 10:27:01 InnoDB: 5.5.35 started; log sequence number 2723867081864 150824 10:27:01 [Note] Server hostname (bind-address): <ip and port here>; 150824 10:27:01 [Note] - <ip here> resolves to <ip here>; 150824 10:27:01 [Note] Server socket created on IP: <ip here>. InnoDB: Error: tried to read 16384 bytes at offset 70 1898921984. InnoDB: Was only able to read -1. 150824 10:27:01 InnoDB: Operating system error number 5 in a file operation. InnoDB: Error number 5 means 'Input/output error'. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html InnoDB: File operation call: 'read'. InnoDB: Cannot continue operation.
EDIT 2: based on comment https://serverfault.com/a/716471/306555, I opened a ticket for replacing the disk and got it replaced and RAID rebuilt. The output from hpacucli looks like:-
physicaldrive 2I:1:1 (port 2I:box 1:bay 1, SATA, 3 TB, OK) physicaldrive 2I:1:2 (port 2I:box 1:bay 2, SATA, 3 TB, OK)
So the Predictive Failure is gone. However, MySQL kept giving me read errors so I ran smartctl long and short test again. While the short test passed, the long one failed with a read error:-
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: read failure 90% 14393 625116232 # 2 Short offline Completed without error 00% 14392 -
I also checked syslog and noticed that every time MySQL tries to start, there's this error in the syslog
Aug 25 14:23:41 kernel: [ 1603.911185] sd 6:0:0:1: [sda] Unhandled sense code Aug 25 14:23:41 kernel: [ 1603.911186] sd 6:0:0:1: [sda] Result: hostbyte=invalid driverbyte=DRIVER_SENSE Aug 25 14:23:41 kernel: [ 1603.911188] sd 6:0:0:1: [sda] Sense Key : Medium Error [current] Aug 25 14:23:41 kernel: [ 1603.911190] sd 6:0:0:1: [sda] Add. Sense: Unrecovered read error Aug 25 14:23:41 kernel: [ 1603.911192] sd 6:0:0:1: [sda] CDB: Read(10): 28 00 46 a2 d5 a0 00 00 08 00
What would that indicate ? (looks like its a bad sector on the disk?) If that's the case, is there a way to fix that ?