1

I was given task to build large SW RAID 50 array on debian 9. Since now i was using mainly HW controllers to do so, which managed a lot of by them selfs.

I have few question now I managed to build desired number of RAID 5 arrays and then put them to RAID 0. This is my build order

3times
mdadm --create /dev/md9 -v --raid-devices=4 --level=5 /dev/sda[ghij]

first question - the raid5 arrays per cat /proc/mdstat are now RECOVERING, not shure why, or if its like initialization, since the disks are empty and iam not trying to recover anything. Do i need to wait before proceeding?

mdadm --create /dev/md22 -v --raid-devices=3 --level=0 /dev/md1 /dev/md2 /dev/md3
mkfs.ext4 -F /dev/md22 

second question - should i do filesystem also for each RAID5 array?

mkdir -p /mnt/md22

mount /dev/md22 /mnt/md22

third question - here the system got stuck, it did not mount the RAID50 or RAID0 with RAID5 arrays to say, not sure why, mabye its connected to recovering state of raid5 or missing filesystem in them? Everything went well up to this point, which is sad since its last step.

edit after 12hours: Its a pool of 3x4 RAID5 put together with RAID0, this is made 7 times, totally its 7xRAID0 over 21xRAID5 over 84 disks. This system is given not invented by by me, cant change it dmesg gave me hint what was wrong, thanks, now its OK the mount went fine out of nowhere today

previously:

[ 4603.087139] EXT4-fs (md22): error loading journal

I think i have another problem with few of the physical disks :/

[ 5502.746501] blk_update_request: critical medium error, dev sds, sector 339957760
[ 5502.746602] raid5_end_read_request: 119 callbacks suppressed
[ 5502.746613] md/raid:md5: read error not correctable (sector 339957760 on sds).
[ 5502.746632] md/raid:md5: read error not correctable (sector 339957768 on sds).
[ 5502.746639] md/raid:md5: read error not correctable (sector 339957776 on sds).
[ 5502.746643] md/raid:md5: read error not correctable (sector 339957784 on sds).
[ 5502.746653] md/raid:md5: read error not correctable (sector 339957792 on sds).
[ 5502.746663] md/raid:md5: read error not correctable (sector 339957800 on sds).
[ 5502.746667] md/raid:md5: read error not correctable (sector 339957808 on sds).
[ 5502.746671] md/raid:md5: read error not correctable (sector 339957816 on sds).
[ 5502.746674] md/raid:md5: read error not correctable (sector 339957824 on sds).
[ 5502.746681] md/raid:md5: read error not correctable (sector 339957832 on sds).
[ 5541.012635] sd 0:0:26:0: [sds] tag#27 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 5541.012661] sd 0:0:26:0: [sds] tag#27 Sense Key : Medium Error [current] 
[ 5541.012667] sd 0:0:26:0: [sds] tag#27 Add. Sense: Unrecovered read error
[ 5541.012676] sd 0:0:26:0: [sds] tag#27 CDB: Read(16) 88 00 00 00 00 00 14 6a 71 78 00 00 04 00 00 00
[ 5541.012681] blk_update_request: critical medium error, dev sds, sector 342520184
[ 5541.012780] raid5_end_read_request: 118 callbacks suppressed
[ 5541.012786] md/raid:md5: read error not correctable (sector 342520184 on sds).
[ 5541.012795] md/raid:md5: read error not correctable (sector 342520192 on sds).
[ 5541.012799] md/raid:md5: read error not correctable (sector 342520200 on sds).
[ 5541.012803] md/raid:md5: read error not correctable (sector 342520208 on sds).
[ 5541.012807] md/raid:md5: read error not correctable (sector 342520216 on sds).
[ 5541.012810] md/raid:md5: read error not correctable (sector 342520224 on sds).
[ 5541.012814] md/raid:md5: read error not correctable (sector 342520232 on sds).
[ 5541.012817] md/raid:md5: read error not correctable (sector 342520240 on sds).
[ 5541.012821] md/raid:md5: read error not correctable (sector 342520248 on sds).
[ 5541.012824] md/raid:md5: read error not correctable (sector 342520256 on sds).
[ 5544.876259] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[ 5544.876268] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[ 5544.876273] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[ 5544.876276] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[ 5544.876341] sd 0:0:26:0: [sds] tag#27 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 5544.876349] sd 0:0:26:0: [sds] tag#27 Sense Key : Medium Error [current] 
[ 5544.876354] sd 0:0:26:0: [sds] tag#27 Add. Sense: Unrecovered read error
[ 5544.876362] sd 0:0:26:0: [sds] tag#27 CDB: Read(16) 88 00 00 00 00 00 14 6a 75 10 00 00 00 08 00 00
[ 5544.876367] blk_update_request: critical medium error, dev sds, sector 342521104
[ 8705.323083] perf: interrupt took too long (2505 > 2500), lowering kernel.perf_event_max_sample_rate to 79750
[18909.385399] sd 0:0:30:0: [sdac] tag#454 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[18909.385427] sd 0:0:30:0: [sdac] tag#454 Sense Key : Medium Error [current] 
[18909.385434] sd 0:0:30:0: [sdac] tag#454 Add. Sense: Unrecovered read error
[18909.385442] sd 0:0:30:0: [sdac] tag#454 CDB: Read(16) 88 00 00 00 00 00 3c 7e d0 68 00 00 04 00 00 00
[18909.385458] blk_update_request: critical medium error, dev sdac, sector 1014943848
[18909.385559] raid5_end_read_request: 119 callbacks suppressed
[18909.385569] md/raid:md8: read error not correctable (sector 1014943848 on sdac).
[18909.385580] md/raid:md8: read error not correctable (sector 1014943856 on sdac).
[18909.385588] md/raid:md8: read error not correctable (sector 1014943864 on sdac).
[18909.385592] md/raid:md8: read error not correctable (sector 1014943872 on sdac).
[18909.385595] md/raid:md8: read error not correctable (sector 1014943880 on sdac).
[18909.385601] md/raid:md8: read error not correctable (sector 1014943888 on sdac).
[18909.385611] md/raid:md8: read error not correctable (sector 1014943896 on sdac).
[18909.385619] md/raid:md8: read error not correctable (sector 1014943904 on sdac).
[18909.385623] md/raid:md8: read error not correctable (sector 1014943912 on sdac).
[18909.385626] md/raid:md8: read error not correctable (sector 1014943920 on sdac).
5
  • 2
    Did you get any kernel warnings/errors in your display or logs or when you run dmesg?
    – HBruijn
    Aug 13, 2018 at 16:29
  • 1
    I know it doesn't answer your questions, but have you considered ZFS? Setting up your system with three, four disk raidz1 vdevs would give you about the same results more or less.
    – Zoredache
    Aug 13, 2018 at 17:18
  • Its a pool of 3x4 RAID5 put together with RAID0, this is made 7 times, totally its 7xRAID0 over 21xRAID5 over 84 disks. This system is given not thought about by me, cant help it
    – J B
    Aug 14, 2018 at 7:31
  • MD RAID 5 arrays always start out in RECOVERING because the entire contents of each disk must be initialized, or else you'd have parity errors everywhere but the blocks written by your application. You do not need to wait for this to finish. The kernel manages I/O correctly while the array is recovering. Aug 14, 2018 at 21:06
  • Those errors sure look like sds and sdac are failing. Time to get out smartctl and see what's been reported. Aug 14, 2018 at 21:07

0

You must log in to answer this question.

Browse other questions tagged .