mpt2sas1: log_info(0x31120303): Disks dropped from RAID

Question

In my syslog I get loads of:

Mar  1 11:20:07 franklin kernel: [72947.878407] Waking error handler thread
Mar  1 11:20:07 franklin kernel: [72947.878415] Error handler scsi_eh_1 waking up
Mar  1 11:20:07 franklin kernel: [72947.878834] scsi_eh_1: flush finish cmd: ffff8806d5568980
Mar  1 11:20:07 franklin kernel: [72947.878871] scsi_restart_operations: waking up host to restart
Mar  1 11:20:07 franklin kernel: [72947.878888] Error handler scsi_eh_1 sleeping
Mar  1 11:20:07 franklin kernel: [72947.878922] scsi_block_when_processing_errors: rtn: 1
Mar  1 11:20:07 franklin kernel: [72947.883450] Waking error handler thread
Mar  1 11:20:07 franklin kernel: [72947.883462] Error handler scsi_eh_1 waking up
Mar  1 11:20:07 franklin kernel: [72947.883887] scsi_eh_1: flush finish cmd: ffff8806d57c0280
Mar  1 11:20:07 franklin kernel: [72947.883927] scsi_restart_operations: waking up host to restart
Mar  1 11:20:07 franklin kernel: [72947.883965] scsi_block_when_processing_errors: rtn: 1
Mar  1 11:20:07 franklin kernel: [72947.883979] Error handler scsi_eh_1 sleeping
Mar  1 11:20:07 franklin kernel: [72947.889556] Waking error handler thread
Mar  1 11:20:07 franklin kernel: [72947.889569] Error handler scsi_eh_1 waking up
Mar  1 11:20:07 franklin kernel: [72947.890015] scsi_eh_1: flush finish cmd: ffff8806d57c0280
Mar  1 11:20:07 franklin kernel: [72947.890052] scsi_restart_operations: waking up host to restart
Mar  1 11:20:07 franklin kernel: [72947.890070] Error handler scsi_eh_1 sleeping
Mar  1 11:20:07 franklin kernel: [72948.569299] mpt2sas1: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303)
Mar  1 11:20:07 franklin kernel: [72948.569312] mpt2sas1: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303)
Mar  1 11:20:07 franklin kernel: [72948.569323] mpt2sas1: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303)
Mar  1 11:20:07 franklin kernel: [72948.569332] mpt2sas1: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303)
Mar  1 11:20:07 franklin kernel: [72948.569342] mpt2sas1: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303)
Mar  1 11:20:07 franklin kernel: [72948.569351] mpt2sas1: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303)
Mar  1 11:20:07 franklin kernel: [72948.569360] mpt2sas1: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303)
Mar  1 11:20:07 franklin kernel: [72948.569370] mpt2sas1: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303)
Mar  1 11:20:07 franklin kernel: [72948.569379] mpt2sas1: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303)

I have enabled extra logging using:

scsiloglev -w -e 7 -t 7 -s 7 -ml 0 -mc 0 -ll 7 -lc 7 -hl 0 -hc 0 -i 0

I have adjusted SCSI timeout:

parallel echo 300 '>' {} ::: /sys/block/sd*[a-z]/device/timeout

And set TLER to 7 seconds:

parallel smartctl -l scterc,70,70 {} ::: /dev/sd*[a-z]

I have changed the controller with an identical controller (SAS2008), reseated all cables, exchanged external SAS cables, reseated all disks. I have no problem reading disks individually using 'dd', but when used in a RAID6 the disks drop offline often.

# uname -a
Linux franklin 3.2.0-0.bpo.4-amd64 #1 SMP Debian 3.2.35-2~bpo60+1 x86_64 GNU/Linux

Anything else I should try before posting on LKML?

Answer 1

These mpt2sas log messages with this log_info usually indicate something bad along the SAS network path. That is, a bad cable or a bad connector. If you have different cables or different caddies to test or even alternative disks that would be a good idea. I've seen these errors coming from bad disks at times. You can try to pinpoint to this by looking at the /sys/class/sas_phy/ hierarchy for invalid_dword files and mapping the affected phy to the component. Notice that the errors will show up on the receiving side so the affected part will be the other side or the cable between them.