Now, granted, issues with RAID rarely come up (at least with the one that I have to deal with in my life), so I'm not the most experienced person in the world. However, I'm not seeing any issues with the status of the array.
I have a 15T array, with five disks. The disks are in a USB 3.0 container. The host system is Ubuntu 14.04 .
When I got up, this morning, I wasn't being able to access files on it. I could hear an irregular pattern of disk activity (like the disks were being overworked, or there was some kind of diagnostics happening).
My /dev/md0 was missing (no longer assembled). My /etc/mdadm/mdadm.conf no longer had an entry for the array (which is totally unexplainable). My /proc/mdstat no longer showed the presence of an array:
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
unused devices: <none>
I ran "sudo mdadm --examine /dev/sd[bcdefghijklmn]1", and everything looks pristine (events match, their statuses are all "clean", and their roles are all active):
mdadm: No md superblock detected on /dev/sdb1.
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 666a93ae:b74a5df0:fb85fdb3:0883f898
Name : dustinbookend:0 (local to host dustinbookend)
Creation Time : Thu Dec 20 01:43:35 2012
Raid Level : raid5
Raid Devices : 5
Avail Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
Array Size : 11721054208 (11178.07 GiB 12002.36 GB)
Used Dev Size : 5860527104 (2794.52 GiB 3000.59 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 8cea6191:38d7f8f5:136f4724:4df69d84
Update Time : Thu Jul 24 09:46:54 2014
Checksum : 764ebaaf - correct
Events : 4616
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAA ('A' == active, '.' == missing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 666a93ae:b74a5df0:fb85fdb3:0883f898
Name : dustinbookend:0 (local to host dustinbookend)
Creation Time : Thu Dec 20 01:43:35 2012
Raid Level : raid5
Raid Devices : 5
Avail Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
Array Size : 11721054208 (11178.07 GiB 12002.36 GB)
Used Dev Size : 5860527104 (2794.52 GiB 3000.59 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 2b8c6859:16736e05:b9db8265:ffb57172
Update Time : Thu Jul 24 09:46:54 2014
Checksum : 7cda2484 - correct
Events : 4616
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAA ('A' == active, '.' == missing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 666a93ae:b74a5df0:fb85fdb3:0883f898
Name : dustinbookend:0 (local to host dustinbookend)
Creation Time : Thu Dec 20 01:43:35 2012
Raid Level : raid5
Raid Devices : 5
Avail Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
Array Size : 11721054208 (11178.07 GiB 12002.36 GB)
Used Dev Size : 5860527104 (2794.52 GiB 3000.59 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 15d0e44b:a0fd34d2:67f7af22:a53b6906
Update Time : Thu Jul 24 09:46:54 2014
Checksum : 8d41944d - correct
Events : 4616
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAA ('A' == active, '.' == missing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 666a93ae:b74a5df0:fb85fdb3:0883f898
Name : dustinbookend:0 (local to host dustinbookend)
Creation Time : Thu Dec 20 01:43:35 2012
Raid Level : raid5
Raid Devices : 5
Avail Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
Array Size : 11721054208 (11178.07 GiB 12002.36 GB)
Used Dev Size : 5860527104 (2794.52 GiB 3000.59 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 9b53a27c:2302f978:86921e06:5b27a873
Update Time : Thu Jul 24 09:46:54 2014
Checksum : b570a32c - correct
Events : 4616
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAA ('A' == active, '.' == missing)
/dev/sdg1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 666a93ae:b74a5df0:fb85fdb3:0883f898
Name : dustinbookend:0 (local to host dustinbookend)
Creation Time : Thu Dec 20 01:43:35 2012
Raid Level : raid5
Raid Devices : 5
Avail Dev Size : 5860528128 (2794.52 GiB 3000.59 GB)
Array Size : 11721054208 (11178.07 GiB 12002.36 GB)
Used Dev Size : 5860527104 (2794.52 GiB 3000.59 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 52dd0d74:db0fa830:775645af:ebcd7e9a
Update Time : Thu Jul 24 09:46:54 2014
Checksum : 3488a60f - correct
Events : 4616
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAA ('A' == active, '.' == missing)
I tried assembling by hand, which completed after a minute or two:
$ sudo mdadm --assemble /dev/md0 /dev/sd[c-g]1
mdadm: /dev/md0 has been started with 5 drives.
I tried mounting it, my label was missing. When I tried fixing the label or mounting the block device, I got a superblock error:
$ sudo mount /mnt/tower/
[sudo] password for dustin:
mount: special device LABEL=tower does not exist
$ sudo e2label /dev/md0 tower
e2label: Attempt to read block from filesystem resulted in short read while trying to open /dev/md0
Couldn't find valid filesystem superblock.
$ sudo mount /dev/md0 /mnt/tower/
mount: /dev/md0: can't read superblock
When I printed status, none of them show as active:
$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sdc1[0](F) sdg1[5](F) sdf1[3](F) sde1[2](F) sdd1[1](F)
11721054208 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/0] [_____]
A little scared, I stopped the array:
$ sudo mdadm --stop /dev/md0
However, now those disk devices no longer show up in /dev (would I have to remove the array in order for the devices to come back?).
I'd really appreciate some advice, here. I don't understand what's going on. I don't think my disks are bad and I don't think this is a USB-controller problem, but I don't think there are a lot of other things that I can investigate without ignorantly risking my data.
Thanks.