7

Im a part time systems administrator and there is little formal documentation. Our main file server was set up in 2010 by somebody who is not even at the institute any more. The requirement at the time was to have three shared filesystems: /home with quota per user and backup, /usr/remote for large software packages and a /scratch without a quota and no backup. Digging through the disks I found an LVM and a RAID controller with a proprietary software.

The server apparently has 14 disks with 1 TB capacity each. They are linked up in pairs using RAID 1. The RAID controller (Areca Technology Corp. ARC-1280/1280ML 24-Port PCI-Express to SATA II RAID Controller) is capable of RAID 5 and 6 (specs), but somehow that has not been used. Then one pair is used as the root partition for that server, /. The other six pairs are linked into a single LVM group and that group then has the three partitions.

enter image description here

I asked about the reasoning and one of the senior people could only tell me that the person who set it up supposedly knew a fair share of LVM and had really thought about this. But I fail to see the advantages of this setup. I only see disadvantages:

  • We have 12 disks with 1 TB of raw storage, with the pairing up we end up with only 6 TB of actually usable space. Using RAID 5 with all of these disks we would have gotten 11 TB of storage, with RAID 6 it would have been 10 TB. Even if we took two sets of six disks into RAID 6, then it would still be 8 TB of usuable space.

  • The worst case disk failure is a whole pair, then part of the LVM is lost and I would think that basically the system would be screwed then. The best case disk failure is one of each pair, so that would be seven broken disks. So from a redundancy standpoint we are not better than RAID 5, given the worst case.

With RAID 6 we would have two redundancies and still could use much more space compared to the current setup. So what is the crucial advantage of this that led people to set it up in this way?

| improve this question | | | | |
  • 3
    2010. Replace it with a new server. Really. – Reinstate Monica - M. Schröder Nov 20 '19 at 20:52
  • 2
    There are plans to replace this server with a VM running CentOS 8 and drawing the block devices from our new Ceph pool. It just hasn't happened yet. – Martin Ueding Nov 21 '19 at 11:12
7

With RAID 6 we would have two redundancies and still could use much more space compared to the current setup. So what is the crucial advantage of this that led people to set it up in this way?

It's true, but together with larger capacity and guaranteed 2 disk redundancy, you’ll get much slower performance on random write operations compared to RAID10 (the best setup in your case on my opinion) . For backups – it can be ok, for user shares and another workload is questionable. In addition old HDDs have higher failure risk during RAID5/6 rebuilt.

| improve this answer | | | | |
  • 2
    RAID 10 in our case would mean that we would link the pairs with RAID 0 and then two pairs to a RAID 1? I would assume that throughput on single large files might be higher, but I'd guess that latency would increase. — Our high performance systems in the clusters have RAID 6 with around 20 disks per controller. And then we have a couple of Lustre servers that provide this over Infiniband. Since that is operated by the cluster people I'd guess that they had used RAID 10 instead of RAID 6 if it would be faster? – Martin Ueding Nov 18 '19 at 17:12
  • 1
    raid 10 will be faster ususally as raid 6 due the fact of missing parity – djdomi Nov 18 '19 at 17:27
  • 3
    @Martin Ueding. RAID10 is faster by design. Under faster I mean lower latency and higher IOPS count on write operations. With 20 disks you'll get 10xRAID1 groups united in one RAID0 – batistuta09 Nov 19 '19 at 8:26
  • 2
    Lustre, as a distributed parallel file system, is a different beast. Often those are spread across many nodes with small disk arrays or even single disks. Not necessarily dozens of disks per array. – John Mahowald Nov 19 '19 at 15:59
4

Premise: I agree that a RAID6 array would have made a lot of sense. However, with so many relatively big disks, I strongly suggest avoiding RAID5 due to high chances of second drive failure during a rebuild.

However, RAID6 comes with significant performance penalty especially during a rebuild. For this very reason, in performance critical setups I generally use RAID10, or stripe over mirrored pairs (note: RAID 0+1, or mirror over striped pairs, should be avoided due to lower resilience).

From the specifications you posted above, it seems that the Areca controller does not support RAID10 or other nested RAID modes. If the old sysadmin decided to avoid RAID6 due to its performance and long rebuild time implications, concatenating the individual arrays in a bigger volume group was the simpler approach.

That said, it is not the better performing approach: as the arrays are just concatenated, the low queue depth streaming performance (ie: single process sequential read/write) is going to be bound to that of a single mirrored pair. To avoid that, another software-based RAID0 layer should be put on top of the mirrored pairs - either via plain md or the newer striped LVM setup.

| improve this answer | | | | |
3

We can only speculate, but this setup seems entirely reasonable to me. The pro's I can see include -

  1. Disaster recovery - even though its a proprietary controller its fairly likely a RAID1 disk can be read in another controller if the main controller fails.

  2. Speed - the specs of the controller say nothing about speed for RAID1 or RAID6 - but its probable RAID1 disks will perform significantly faster then other variants for highly scattered workloads as there are no calculations.

  3. Expandability - It may well be he started with a smaller array and just added pairs as time went by. This would be significantly easier then resilvering a disk. (It is a bit strange that all disks are 1tb in this case though)

  4. Higher redundancy and quicker rebuilds in case of certain failures. (Out of curiosity are the disks used in each half of the RAID different models or batches? If so it could show a deliberate thinking of failure modes)

While I would gave put / on the LVM as well, its not unreasonable to keep it separate to allow for easier setup. Also RAID6 was not as common 9 years ago as disks were smaller, so there was less perceived need.

| improve this answer | | | | |
  • Regarding 3: I don't think that the thing got extended. From the logs I gather that they started with the disks as they have been. One of them has been switched against a 2 TB replacement due to failure. — Regarding 4: Looking at the RAID controller all disks are all WDC WD1002FBYS-02A6B0. And the serial numbers are not consecutive, but I cannot spot a pattern. — I just realized that the drives hosting / are just 320 GB of size, so that might be why they were excluded. – Martin Ueding Nov 20 '19 at 9:17
  • 1
    They missed a trick in regard of #4 then. They should be using different models (ideally different makes) of disks in each half of the array to reduce the likelihood of a second failure of a disk at about the same time. – davidgo Nov 20 '19 at 17:51
2

nice diagram! :-)

Yea, that does seem a little odd. Without specifics on the make/model we don't know if this was done due to a limitation of the hardware or something, but if ont I'd be surprised it it was due to anything other than a misunderstanding of the available RAID options or a lack of understanding of how to set it up. Maybe they knew LVM really well and just stuck to what they knew.

I'd usually recommend relying on hardware raid where its available for stability and performance reasons.

| improve this answer | | | | |
  • 3
    I have added the make and model of the RAID controller, as well as a link to the spec. It supposedly supports 0, 1, 1E, 3, 5, 6 and a proprietary JBOD RAID level. – Martin Ueding Nov 18 '19 at 17:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.