I inherited a NAS server that is setup with 5 RAID1 soft arrays grouped together into a XFS volume group.
Whoever built the 5th md device created it without a Linux Auto RAID partition (Looks like they did a mdadm --create using the raw disk (i.e. /dev/sdj /dev/sdk). It had been working fine until now, but today the entire /dev/md5 array disappeared.
The /dev/sdj drive appears to be in the process of failing. Buffer I/O error on /dev/sdj logical block 0 Buffer I/O error on /dev/sdj logical block 1 Buffer I/O error on /dev/sdj logical block 2 Buffer I/O error on /dev/sdj logical block 3
Normally I would expect the RAID to fail the device, but keep the array up with the 2nd drive. When I cat /proc/mdstat however, my md5 device is gone. I suspect this is because those two drives did not have a Auto RAID partition, but I'm not sure.
I've tried to re-create the md5 array using
mdadm --create /dev/md5 --level=1 --raid-devices=2 /dev/sdj /dev/sdk
but it says sdj is already part of a RAID device
What is odd is the XFS volume group appears to still be working fine - no data is lost as far as I can tell, and df still shows all the space being available. Could it be XFS still sees the /dev/sdk drive and can successfully write to it? Both sdj and sdk show up with a fdisk -l.
My questions are:
- Can I safely replace the /dev/sdj drive without messing up the (working but fragile) XFS volume?
- How can I recover/rebuild the md5 Array if mdstat says it does not exist, but mdadm says it does?
- If I go and add a Linux Auto RAID partition to the remaining good drive in this array, will that corrupt the data already on it?
- How do you verify data integrity with XFS with ? (to ensure there really is no data loss)
Output of pvscan:
pvscan /dev/sdj: read failed after 0 of 4096 at 0: Input/output error /dev/sdj: read failed after 0 of 4096 at 2000398843904: Input/output error PV /dev/sdd2 VG VolGroup00 lvm2 [74.41 GB / 0 free] PV /dev/md2 VG dedvol lvm2 [931.51 GB / 0 free] PV /dev/md3 VG dedvol lvm2 [931.51 GB / 0 free] PV /dev/md0 VG dedvol lvm2 [931.51 GB / 0 free] PV /dev/md4 VG dedvol lvm2 [931.51 GB / 0 free] PV /dev/sdj VG dedvol lvm2 [1.82 TB / 63.05 GB free] Total: 6 [5.53 TB] / in use: 6 [5.53 TB] / in no VG: 0 [0 ]
Disk /dev/sdj: 2000.3 GB, 2000398934016 bytes 255 heads, 63 sectors/track, 243201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk /dev/sdj doesn't contain a valid partition table Disk /dev/sdk: 2000.3 GB, 2000398934016 bytes 255 heads, 63 sectors/track, 243201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk /dev/sdk doesn't contain a valid partition table
mdadm --misc -Q /dev/sdj /dev/sdj: is not an md array /dev/sdj: No md super block found, not an md component. mdadm --misc -Q /dev/sdk /dev/sdk: is not an md array /dev/sdk: device 0 in 2 device undetected raid1 /dev/md5. Use mdadm --examine for more detail.
mdadm --examine /dev/sdk /dev/sdk: Magic : a92b4efc Version : 0.90.00 UUID : 25ead1e4:9ab7f998:73875d59:48b17be5 Creation Time : Fri Nov 26 21:10:49 2010 Raid Level : raid1 Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB) Array Size : 1953514496 (1863.02 GiB 2000.40 GB) Raid Devices : 2 Total Devices : 2 Preferred Minor : 5 Update Time : Sat Mar 26 07:43:52 2011 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Checksum : 35a405cb - correct Events : 5720270 Number Major Minor RaidDevice State this 0 8 144 0 active sync /dev/sdj 0 0 8 144 0 active sync /dev/sdj 1 1 8 160 1 active sync /dev/sdk