Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a RAID 5 array (with LVM on top) on a CentOS 6 box. The array itself is an LSI StorageTek disk shelf with 14 drives connected to the server with a fibre channel cable.

After rebooting the machine the RAID array won't come back up. Disk Utility in Gnome states that it is "Not running, partially assembled".

I ran mdadmin --assemble --scan, which said:

mdadm: /dev/md/:storagetek-1_0 assembled from 1 drive - not enough to start the array.
mdadm: No arrays found in config file or automatically

/proc/mdstat says:

Personalities : [raid6] [raid5] [raid4] 
md127 : inactive sdf1[7] sdi1[4] sde1[8] sdj1[3] sdc1[10] sdg1[6] sdd1[9] sdn1[12] sdb1[11] sdm1[0] sda1[14] sdk1[2]
      860171694 blocks super 1.2
unused devices: <none>

I'm pretty new to managing RAID arrays on Linux (could you guess?) so I've reached the limit of my very limited knowledge on the subject.

I'm optimistically hoping that it's in the process of being rebuilt, but from what I've seen I doubt it.

Please can somebody give me a hint on how to fix it?

share|improve this question

2 Answers 2

It's not in the process of rebuilding; if so, /proc/mdstat would tell you, and not say inactive.

Looking at the message,

mdadm: /dev/md/:storagetek-1_0 assembled from 1 drive - not enough to start the array.

I'd guess that the machine is only seeing one of the drives on the shelf. (Or perhaps they got renumbered, but I think mdadm is smart enough to deal with that.) I don't know much about Linux FC support and such, but perhaps there's some tool which shows you what it's seeing?

Also, definitely run dmesg to see if there's anything about the tray, drives, RAID, etc. in the kernel log.

share|improve this answer
    
pastebin.com/NFXKxvw8 Line 240 seems to suggest that two drives in the array have failed, disks #1 and #5 if I'm understanding it correctly. Strange that all of the LEDs on the disk shelf are green. Also rather annoying as I've only just bought the thing (albeit second hand). –  Matt Ralston Dec 14 '11 at 11:09
    
I agree. Lines 215-226 show 12 "operational" disks: [a-n] excluding h and l. On 194-199, it says "kicking non-fresh sdh1 from array!" (and same for sdl.) Googling around pointed to this post and also mdadm --examine --detail. You could try putting the array together with mdadm --assemble --force /dev/md127 /dev/sd{a,b,c,...,n}1 or something like this form post –  jon Dec 16 '11 at 18:13
    
Before you start doing things, I'd definitely take a look at the SMART status for the disks. First install smartctl (package smartmontools in CentOS 5.5.) Useful commands: smartctl -s on /dev/sda to turn SMART on for /dev/sda; smartctl -s on -o on -S on /dev/sda to turn on automatic testing; smartctl -t {short | long} /dev/sda to run a self-test; smartctl -a /dev/sda to show information. Most of the output is (relatively) self-explanatory. For interpreting the Attributes table, "VALUE < THRESH" is bad. One note: SMART saying your drive is good only means it hasn't found any errors. –  jon Dec 16 '11 at 18:24

Thanks for the info. Just got back to this after Christmas. SMART tests suggest that /dev/sdl in my array has failed, but that /dev/sdh is ok. I guess that means that I need to replace /dev/sdl. Is there a way to make the light on the drive flash or something so that I can ensure I get the right one? By way of some background, this problem started when I hot-removed one drive to get the serial number off it to order some spares (ok, bad idea with hindsight, but I assumed they were hot swappable). I think what probably happened is that I removed /dev/sdh and that /dev/sdl had already failed previously which left the array with 2 missing nodes so it failed. As /dev/sdh appears to be ok, what's the right command to re-add it to the array?

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.