I have a raid 5 mdadm raid array set up with 6 drives and a hot spare on an Ubuntu 11 server. There are a couple samba shares on the drive, and until today, they were working well.
A couple hours ago users started noticing their shares were crawling or not connecting at all, taking many minutes to list the files present. Trying to copy files would freeze the connection most of the time and eventually disconnect them. I was able to browse the shares on the mounted directory fine through ssh, but samba was definitely having issues. I tried restarting samba to no effect.
I ran mdadm --detail /dev/md2 and... nothing. It didn't output anything or return my prompt, and I had to control-c it to get my prompt back. /proc/mdstat was also empty. But for some reason, I could still browse the mounted raid array and everything looked fine. Looking back I should have tried adding and removing files through the terminal...
Checking the process monitor showed a bunch of smbd processes for each user hanging in the D state, and I couldn't stop them with a kill command. I hadn't seen something like this, and with mdadm giving nothing useful I tried to reboot the server. That also hung. I crossed my fingers and told the datacenter guy to hit the hard reset.
In the end, the raid is rebuilding fine and all drives are up. But I'm still not sure what would cause mdadm to freeze up like that, disconnect all the samba connections, and be unresponsive.
I'm pretty new to all this so was hoping to get some help debugging the problem from those of you who might have seen similar issues before. Where would you look first?
EDIT:: Following ACase's advice, here is some more diagnostic information:
The filesystem on /dev/md2 (the RAID drive in question) is ext3
Here is my kernel info
2.6.35-22-server #33-Ubuntu SMP Sun Sep 19 20:48:58 UTC 2010 x86_64 GNU/Linux
Looking in /var/log/messages reveals that before the restart I was having a bunch of these errors (maybe 15 every 3 seconds) during the timeframe when the drives were inaccessible through samba:
kernel: [17343195.826943] mptbase: ioc0: LogInfo(0x31123000): Originator={PL}, Code={Abort}, SubCode(0x3000)
which, through some googling, appears like it might be related to SMART results ran with SATA drives through an SAS controller. The server is a dell t610 with SAS 6/iR Integrated, so this very well might be what is causing my problem - MDADM tries to run Smart on the drives and then freezes IO with all the errors. Does that sound right? What tests would you run to confirm it? I'd rather not take the whole array down again if possible, since it is being used (prematurely, obviously). That log message stops showing up after reboot, and then samba works again, so I'm pretty sure they are related. No messages show up in between those ones - is there a way to turn on more verbose kernel logging into /var/log/messages that might prove they are SMART related?
Thanks again.
chkconfig --list | grep smart
)?/etc/mdadm.conf
?smartctl -a /dev/sd[a-z]
(assuming the devices in RAID are labeled sd[a-z]) without bringing down your RAID partition?