Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We had a storage server with performance starting to be poor. When I installed Linux's smartutils, it found

Device: /dev/sdm, 2 Offline uncorrectable sectors

Since this was a rather large raid6 array ( 23 x 3TB setup ), I set the drive to faulty, and removed it from the array. I shutdown the server, replaced the drive, and powered it back up. When I powered the server back up, it could not find md0. As an idiot, I tried to recreate the array using

mdadm --create /dev/md0 --level=6 --raid-devices=23 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx

I felt it was wrong, so I set the drive faulty again, removed it, then re-added it.

Later that night, I started getting kernel errors being reported to /var/log/messages.

Jan 11 21:20:05 sv-* kernel: Filesystem "dm-2": XFS internal error xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c. Caller 0xffffffff8833682e

How bad is the corruption, and could it have corrupted actual files too instead of just directories?

Update 1 of 1

uname output

Linux sv-* 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

Centos release 5.5 (Final)

yum list installed|grep xfs

kmod-xfs.x86_64 0.4-2 installed

xfsprogs.x86_64 2.9.4-1.el5.centos installed

xfs_check /dev/lvm-raid/lvm0

bad magic # 0 in btcnt block 0/488807
bad magic # 0 in btcnt block 0/920869
bad directory data magic # 0 for dir ino 228 block 0
no . entry for directory 228
no .. entry for directory 228
bad directory data magic # 0 for dir ino 229 block 0
no . entry for directory 229
no .. entry for directory 229
bad directory data magic # 0 for dir ino 232 block 0
no . entry for directory 232
no .. entry for directory 232
bad directory data magic # 0 for dir ino 237 block 0
no . entry for directory 237
no .. entry for directory 237
bad directory data magic # 0 for dir ino 242 block 0
no . entry for directory 242
no .. entry for directory 242
bad directory data magic # 0 for dir ino 244 block 0
no . entry for directory 244
no .. entry for directory 244
bad directory data magic # 0 for dir ino 246 block 0
no . entry for directory 246
no .. entry for directory 246
bad directory data magic # 0 for dir ino 249 block 0
no . entry for directory 249
no .. entry for directory 249
bad directory data magic # 0 for dir ino 1285 block 0
bad directory data magic # 0 for dir ino 1285 block 1
multiple .. entries in dir 1285
bad directory data magic # 0 for dir ino 1285 block 2
multiple .. entries in dir 1285
bad directory data magic # 0 for dir ino 1285 block 3
multiple .. entries in dir 1285
dir 1285 block 8388609 extra leaf entry b33bb481 775
dir 1285 block 8388609 extra leaf entry b33bb489 7bb
dir 1285 block 8388609 extra leaf entry b33bb491 6e9
... In a previous run, the xfs_check ate through the 8 GB of ram, and 10 GB of swap before it ran out of ram, and died.

share|improve this question
    
Did mdadm complain about existing superblocks when you did the --create? –  Andrew Jan 17 '12 at 2:38
    
This sounds like a volume issue at this point, can you provide a 'uname -ar' and the distro (release) you're using? Are you currently experiencing any data loss? I'd see what xfs_repair shows. –  orbitron Jan 17 '12 at 2:49
    
Andrew, I want to say yes, but I'm not totally sure. –  becomingwisest Jan 17 '12 at 3:01
    
If there's one thing I learned when getting familiar with using the --create command instead of the proper --assemble, it's that RAID5 is resilient against an errant --create - but RAID6 would not be, due to the second backup block using an algorithm other than a straight XOR. Do you have an old copy of /proc/mdstat to verify that you rebuilt in the correct layout? Did you stop the resync from the --create relatively early on in the process? Do you have a backup or other means to validate the data? –  Shane Madden Jan 17 '12 at 3:24
1  
possible duplicate of Recover RAID 5 data after created new array instead of re-using –  Chris S Jan 17 '12 at 3:49
show 1 more comment

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.