1

I have a RAID 5 array (with LVM on top) on a CentOS 6 box. The array itself is an LSI StorageTek disk shelf with 14 drives connected to the server with a fibre channel cable.

After rebooting the machine the RAID array won't come back up. Disk Utility in Gnome states that it is "Not running, partially assembled".

I ran mdadmin --assemble --scan, which said:

mdadm: /dev/md/:storagetek-1_0 assembled from 1 drive - not enough to start the array.
mdadm: No arrays found in config file or automatically

/proc/mdstat says:

Personalities : [raid6] [raid5] [raid4] 
md127 : inactive sdf1[7] sdi1[4] sde1[8] sdj1[3] sdc1[10] sdg1[6] sdd1[9] sdn1[12] sdb1[11] sdm1[0] sda1[14] sdk1[2]
      860171694 blocks super 1.2
unused devices: <none>

I'm pretty new to managing RAID arrays on Linux (could you guess?) so I've reached the limit of my very limited knowledge on the subject.

I'm optimistically hoping that it's in the process of being rebuilt, but from what I've seen I doubt it.

Please can somebody give me a hint on how to fix it?

0

2 Answers 2

0

It's not in the process of rebuilding; if so, /proc/mdstat would tell you, and not say inactive.

Looking at the message,

mdadm: /dev/md/:storagetek-1_0 assembled from 1 drive - not enough to start the array.

I'd guess that the machine is only seeing one of the drives on the shelf. (Or perhaps they got renumbered, but I think mdadm is smart enough to deal with that.) I don't know much about Linux FC support and such, but perhaps there's some tool which shows you what it's seeing?

Also, definitely run dmesg to see if there's anything about the tray, drives, RAID, etc. in the kernel log.

3
  • pastebin.com/NFXKxvw8 Line 240 seems to suggest that two drives in the array have failed, disks #1 and #5 if I'm understanding it correctly. Strange that all of the LEDs on the disk shelf are green. Also rather annoying as I've only just bought the thing (albeit second hand). Dec 14, 2011 at 11:09
  • I agree. Lines 215-226 show 12 "operational" disks: [a-n] excluding h and l. On 194-199, it says "kicking non-fresh sdh1 from array!" (and same for sdl.) Googling around pointed to this post and also mdadm --examine --detail. You could try putting the array together with mdadm --assemble --force /dev/md127 /dev/sd{a,b,c,...,n}1 or something like this form post
    – jade
    Dec 16, 2011 at 18:13
  • Before you start doing things, I'd definitely take a look at the SMART status for the disks. First install smartctl (package smartmontools in CentOS 5.5.) Useful commands: smartctl -s on /dev/sda to turn SMART on for /dev/sda; smartctl -s on -o on -S on /dev/sda to turn on automatic testing; smartctl -t {short | long} /dev/sda to run a self-test; smartctl -a /dev/sda to show information. Most of the output is (relatively) self-explanatory. For interpreting the Attributes table, "VALUE < THRESH" is bad. One note: SMART saying your drive is good only means it hasn't found any errors.
    – jade
    Dec 16, 2011 at 18:24
0

Thanks for the info. Just got back to this after Christmas. SMART tests suggest that /dev/sdl in my array has failed, but that /dev/sdh is ok. I guess that means that I need to replace /dev/sdl. Is there a way to make the light on the drive flash or something so that I can ensure I get the right one? By way of some background, this problem started when I hot-removed one drive to get the serial number off it to order some spares (ok, bad idea with hindsight, but I assumed they were hot swappable). I think what probably happened is that I removed /dev/sdh and that /dev/sdl had already failed previously which left the array with 2 missing nodes so it failed. As /dev/sdh appears to be ok, what's the right command to re-add it to the array?

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .