I have a pair of HP DL320e servers identically configured with 2 x WD Red 6TB drives in a software raid 1 array.
# cat /proc/mdstat Personalities : [raid1] md126 : active raid1 sda3 sdb3 5859876672 blocks super 1.2 [2/2] [UU] bitmap: 0/44 pages [0KB], 65536KB chunk md127 : active raid1 sda2 sdb2 511936 blocks super 1.0 [2/2] [UU] unused devices: <none>
The machines have been installed for about 6 weeks and generally run fine, however on several occasions one drive in the pair disappears from the configuration, the system reports the drive as having been removed:
# mdadm --detail /dev/md126 /dev/md126: Version : 1.0 Creation Time : Thu Feb 12 12:16:13 2015 Raid Level : raid1 Array Size : 511936 (500.02 MiB 524.22 MB) Used Dev Size : 511936 (500.02 MiB 524.22 MB) Raid Devices : 2 Total Devices : 1 Persistence : Superblock is persistent Update Time : Mon Apr 20 15:39:02 2015 State : clean, degraded Active Devices : 1 Working Devices : 1 Failed Devices : 0 Spare Devices : 0 Name : localhost:boot UUID : 60a3e4f7:793fbc3d:3f409985:5f619dea Events : 139 Number Major Minor RaidDevice State 2 8 2 0 active sync /dev/sda2 2 0 0 2 removed
/var/log/messages reports the drive cannot be opened
Apr 20 14:47:00 1 smartd: Device: /dev/sda [SAT], open() failed: No such device
Both drives appear physically OK and are spinning. These machines form a production pair so I have not done much experimentation. The first time it happened I shut down the server and rebooted it, the failed drive remained failed so I shut it down again, removed the drive which looked normal, reinserted it and rebooted, the drive appeared normal and after
mdadm --manage /dev/md126 --add /dev/sda2
... and the similar command for the other filesystem the raid array re-synched and everything returned to normal.
The same thing has happened with both drives on one machine and today the other server did the same thing with one of its drives. Today I shut the server down and removed the power cord to force a complete cold start including iLo, I did not remove the "failed" drive and when the machine came up the device was visible and the mdadm command to add the missing drive back worked as before.
I can't see anything else in the message log or anywhere else. The failure today happened as the system rebooted after I manually did a yum update and reboot, the previous failure happened after a reboot which happened for reasons I cannot explain when I was not in the office.
In the failed state the raid array operates in degraded mode and the system operates normally, I am not suffering service interruptions as a result, but its obviously a worry that this is happening.
I am not familiar with iLO but I can't see anything in the iLO logs except a note that the power was cycled at the moment of the unexplained reboot.
I am running Centos 7, 4Gb ram, Xenon 4 core 3.1GHz with 2 6TB disks. All yum updates have been applied.
Has anyone seen something like this on a DL320e?