0

Last week I replaced a hard drive in a Dell R515 server that has 8 drives in a RAID 10. This server had two rows of hard drives; 4 on top and 4 on bottom and I replaced the 2nd drive on the top row.

In the past, I've always done this while the server is running, but this time I turned the server off before replacing the hard drive.

When I turned the server back on, I was expecting to see a lot of blinking-light-activity on the new hard drive and also on the hard drive that the new hard drive is to be mirroring.

However, this did not occur. So, I decided to just give it some time....

Today, I rebooted this server and upon doing so I noticed that it indicated that the virtual hard disk is degraded.

Also, normally, while the server is operating, all hard drive lights seem to blink at the same time (during file accesses). However, now, only the bottom row (4 of the 8 drives) blink during file access.

To me, it seems like I did the wrong thing, when I turned off the server before replacing the drive and (because I did this), the server didn't rebuild that new hard drive into the RAID 10.

However, it surprises me that the top row of drives are keeping a solid green light with no blinking while the bottom 4 drives are blinking during file accesses.

Did I cause all 4 of the top row of drives to become out of sync with their respective mirror-drives (on the bottom row)?

A few minutes ago, I pulled out the new hot-swap-hard-drive while the server was running, and then put it back in. I can tell by the light activity that the new hard drive is now being synced with the drive under it.

However, the other 3 hard drive on the top row continue to be solid-green-light, with no apparent activity (blinking). Will they start acting normal after the new hard drive is synced, or will I most likely have to pull each of them too before the server's virtual disk will get out of degraded status?

UPDATE: Pulling the drive and putting it back in while the server was running seems to have fixed the issue. Rebooting no longer warns of a degraded virtual drive. The bottom row of drive lights still blink way more frequently than the top row. Perhaps that makes sense during read access (there's no need to read from mirrors, while there is a need to write to them). Perhaps I recall incorrectly about all drives blinking together.

  • 2
    1. Go into the configuration utility and see why the virtual drive is in a degraged state. 2. Install OMSA on the server so that you can access the hardware components from the server OS. – joeqwerty Oct 10 '16 at 19:37
  • I'll have to check if this is available for my server OS (Proxmox Linux). – LonnieBest Oct 11 '16 at 1:02
  • 1
    That's curious behavior for reads... must be a lot of "small" background reads for it not to pull from the mirrors (which is a big help with read performance during heavy I/O) – JimNim Oct 11 '16 at 1:11
  • Good point. It make sense to me now that you can indeed increase IO by reading from the mirrors too. – LonnieBest Oct 11 '16 at 1:14
2

A word of wisdom I've heard from storage veterans: "Don't troubleshoot with the blinky lights"

The status LEDs can help supplement other data you have available, but they should never be your sole indicator of what's going on unless you have no other option.

+1 on Joe's comment - don't try to fly blind, get OMSA installed (find "OpenManage Server Administrator Managed Node" on http://downloads.dell.com/published/pages/poweredge-r515.html for your OS), and reference the user guide for info on viewing status of the virtual disk and hard drives. You can also export an event log to view what may have occurred leading up to the drive problem, and after power-up - this may give you some more insight into what actually went wrong if it's not already obvious from other data shown in OMSA.

Did I cause all 4 of the top row of drives to become out of sync with their respective mirror-drives (on the bottom row)?

Quite possibly. That wouldn't be considered "normal behavior", but it may be a side effect of not having followed the documented procedure for drive replacement (or a combination of that and old/buggy controller firmware). Stick with hot-swapping in the future and you'll be a lot better off.

  • I'm running Proxmox Linux on this server. I will have to check if "OpenManage Server Administrator Managed Node" can be installed. – LonnieBest Oct 11 '16 at 1:01
  • 1
    Wikipedia says that's Debian-based with a modified RHEL kernel, so it MIGHT work? If not, you could try using MegaCLI from LSI (the OEM of Dell's RAID controllers). You'll have to do some reading on how to use it though. – JimNim Oct 11 '16 at 1:07
  • 1
    Looks like others have gotten it to work: forum.proxmox.com/threads/dell-openmanage-installation.13145 – JimNim Oct 11 '16 at 1:09
  • Thanks for all the help JimNim. I will try it out. Read my update (above). – LonnieBest Oct 11 '16 at 1:09

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.