3

I have a raid 5 mdadm raid array set up with 6 drives and a hot spare on an Ubuntu 11 server. There are a couple samba shares on the drive, and until today, they were working well.

A couple hours ago users started noticing their shares were crawling or not connecting at all, taking many minutes to list the files present. Trying to copy files would freeze the connection most of the time and eventually disconnect them. I was able to browse the shares on the mounted directory fine through ssh, but samba was definitely having issues. I tried restarting samba to no effect.

I ran mdadm --detail /dev/md2 and... nothing. It didn't output anything or return my prompt, and I had to control-c it to get my prompt back. /proc/mdstat was also empty. But for some reason, I could still browse the mounted raid array and everything looked fine. Looking back I should have tried adding and removing files through the terminal...

Checking the process monitor showed a bunch of smbd processes for each user hanging in the D state, and I couldn't stop them with a kill command. I hadn't seen something like this, and with mdadm giving nothing useful I tried to reboot the server. That also hung. I crossed my fingers and told the datacenter guy to hit the hard reset.

In the end, the raid is rebuilding fine and all drives are up. But I'm still not sure what would cause mdadm to freeze up like that, disconnect all the samba connections, and be unresponsive.

I'm pretty new to all this so was hoping to get some help debugging the problem from those of you who might have seen similar issues before. Where would you look first?


EDIT:: Following ACase's advice, here is some more diagnostic information:

The filesystem on /dev/md2 (the RAID drive in question) is ext3

Here is my kernel info

2.6.35-22-server #33-Ubuntu SMP Sun Sep 19 20:48:58 UTC 2010 x86_64 GNU/Linux

Looking in /var/log/messages reveals that before the restart I was having a bunch of these errors (maybe 15 every 3 seconds) during the timeframe when the drives were inaccessible through samba:

kernel: [17343195.826943] mptbase: ioc0: LogInfo(0x31123000): Originator={PL}, Code={Abort}, SubCode(0x3000)

which, through some googling, appears like it might be related to SMART results ran with SATA drives through an SAS controller. The server is a dell t610 with SAS 6/iR Integrated, so this very well might be what is causing my problem - MDADM tries to run Smart on the drives and then freezes IO with all the errors. Does that sound right? What tests would you run to confirm it? I'd rather not take the whole array down again if possible, since it is being used (prematurely, obviously). That log message stops showing up after reboot, and then samba works again, so I'm pretty sure they are related. No messages show up in between those ones - is there a way to turn on more verbose kernel logging into /var/log/messages that might prove they are SMART related?

Thanks again.

5
  • What make/model are the disks in the array? Nov 7, 2011 at 22:41
  • They are sata 2tb drives - 2 hitachi F30ee7ud's and 4 samsung hd2040ui's Nov 7, 2011 at 22:57
  • Are you running smartd (chkconfig --list | grep smart)? Nov 10, 2011 at 0:01
  • I don't think mdadm runs smart to check the disk by default, but it may vary on systems. Can you post your /etc/mdadm.conf? Nov 10, 2011 at 0:02
  • Can you run smartctl -a /dev/sd[a-z] (assuming the devices in RAID are labeled sd[a-z]) without bringing down your RAID partition? Nov 10, 2011 at 0:10

1 Answer 1

2

Look for errors in /var/log/messages or /var/log/kernel. It sounds like the kernel stopped being able to write and/or read to disk. That would explain why it wouldn't reboot nicely.

  • What disk format are you using (ext2,ext3, ext4, xfs, etc.)? Journaled?
  • What kernel are you using? Check to see if there are any kernel bugs for this.
  • When this occurs check to see which partitions (md[0-9]) are readable/writable
  • Use the hdparm utility to check disk I/O speeds and settings are set appropriately

I would generally suggest you run an fsck on your filesystem after this type of occurrence.

In addition Linux has a couple reboot options that will allow you to ignore certain disk issues and force the system to reboot without having to call your DC guy to hard reset it (in order of least to greatest severity):

   -f     Force halt or reboot, don’t call shutdown(8).

   -n     Don’t sync before reboot or halt. Note that the kernel and stor-
          age drivers may still sync.

These are both safer options than a hard reset.

[Edit #1]:

Check output from smartctl -a /dev/sd[a-z] to see if any disks are having problems.

[Edit #2]:

I'd suggest scheduling some downtime and upgrading the firmware. It tends to fix a lot of bugs. Specifically the SAS controller and BIOS. Maybe others if they suggest them.

In addition since this is an t610 does it have a DRAC interface? You can often see hardware related logs there if there was a hardware fault.

3
  • Thanks for the advice, it was really helpful. I modified the question with more information that I think is very close to the solution... I'm hesitant to mark this as the answer though since it doesn't fix the problem, just my incompetence ;) Nov 9, 2011 at 20:47
  • See edits above. Nov 10, 2011 at 0:28
  • I was able to run smartctl on each of the drives in the raid array without any problems. My mdadm.conf doesn't specify anything about SMART testing. To help solve this issue I am setting up the iDRAC card next week to hopefully find if there is a hardware issue, but Since I can't get this to reproduce I am marking this answer as accepted. I'll report back if I find anything specific Nov 12, 2011 at 6:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .