1

I have a pair of HP DL320e servers identically configured with 2 x WD Red 6TB drives in a software raid 1 array.

# cat /proc/mdstat
Personalities : [raid1] 
md126 : active raid1 sda3[2] sdb3[1]
      5859876672 blocks super 1.2 [2/2] [UU]
      bitmap: 0/44 pages [0KB], 65536KB chunk

md127 : active raid1 sda2[3] sdb2[2]
      511936 blocks super 1.0 [2/2] [UU]

unused devices: <none>

The machines have been installed for about 6 weeks and generally run fine, however on several occasions one drive in the pair disappears from the configuration, the system reports the drive as having been removed:

# mdadm --detail /dev/md126
/dev/md126:
        Version : 1.0
  Creation Time : Thu Feb 12 12:16:13 2015
     Raid Level : raid1
     Array Size : 511936 (500.02 MiB 524.22 MB)
  Used Dev Size : 511936 (500.02 MiB 524.22 MB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Mon Apr 20 15:39:02 2015
          State : clean, degraded 
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : localhost:boot
           UUID : 60a3e4f7:793fbc3d:3f409985:5f619dea
         Events : 139

    Number   Major   Minor   RaidDevice State
       2       8        2        0      active sync   /dev/sda2
       2       0        0        2      removed

/var/log/messages reports the drive cannot be opened

Apr 20 14:47:00 1 smartd[983]: Device: /dev/sda [SAT], open() failed: No such device

Both drives appear physically OK and are spinning. These machines form a production pair so I have not done much experimentation. The first time it happened I shut down the server and rebooted it, the failed drive remained failed so I shut it down again, removed the drive which looked normal, reinserted it and rebooted, the drive appeared normal and after

mdadm --manage /dev/md126 --add /dev/sda2

... and the similar command for the other filesystem the raid array re-synched and everything returned to normal.

The same thing has happened with both drives on one machine and today the other server did the same thing with one of its drives. Today I shut the server down and removed the power cord to force a complete cold start including iLo, I did not remove the "failed" drive and when the machine came up the device was visible and the mdadm command to add the missing drive back worked as before.

I can't see anything else in the message log or anywhere else. The failure today happened as the system rebooted after I manually did a yum update and reboot, the previous failure happened after a reboot which happened for reasons I cannot explain when I was not in the office.

In the failed state the raid array operates in degraded mode and the system operates normally, I am not suffering service interruptions as a result, but its obviously a worry that this is happening.

I am not familiar with iLO but I can't see anything in the iLO logs except a note that the power was cycled at the moment of the unexplained reboot.

I am running Centos 7, 4Gb ram, Xenon 4 core 3.1GHz with 2 6TB disks. All yum updates have been applied.

Has anyone seen something like this on a DL320e?

1
  • I think it is a hardware problem, more exactly a communication problem between your hard disk and the disk controller of your motherboard. Alternatively it can be a power supply problem. Anyways, a testing with different drive, cable, plugging in differently, etc, the classical hw troubleshooting will probably help.
    – peterh
    Apr 20 '15 at 17:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.