5

If disks have 512-byte physical sectors, and you have 10 disks using RAID 50 with a 1MB stripe-size how does that work at the disk level?

Correct me if I'm wrong, but conceptually, there would be 2 spans each consisting of a RAID-5 array of 5 disks, one mirrored to the other. Therefore, a "stripe" would consist of 4x256KB chunks of data, plus a single 256KB of parity data per stripes? or does a "stripe" include the parity?

What if you consider a 12-disk RAID 10 array? There would be 6 mirrored pairs of disks, with striping over those mirrors. So, for a 1MB stripe size, the stripe would be divided by 6, for 174,762.666 bytes per-disk, which works out to 341.333 physical sectors per stripe. Is that really 342 physical sectors per stripe?

For those who wonder why I'm asking; I am attempting to determine the most efficient number of disks relative to the type of RAID, with the best stripe size.


Also, I have seen https://en.wikipedia.org/wiki/Nested_RAID_levels prior to asking this question. In fact, I've done a ton of work looking for low-level design details on a vast multitude of SCSI / SAS / RAID / SAN vendor sites, and have not seen anything that talks about the actual on-disk format of stripes. Stripes are only talked about at a highly conceptual level, which is fine, but doesn't really answer the question.

  • 6
    en.wikipedia.org/wiki/Nested_RAID_levels - pooh, that was hard to find. – ott-- May 17 '16 at 20:55
  • 1
    I don't think sysadmins/engineers are concerned with this level of detail anymore. Especially with the prevalence of SSDs, the abundance of hardware RAID solutions with read/write cache and improvements in software-defined-storage and general tiering/caching options. – ewwhite May 17 '16 at 21:48
  • 1
    I happen to know that OP is using spinnies on his server and will be running databases on it. – ConcernedOfTunbridgeWells May 24 '16 at 16:07
5
+200

You will find all relevant details here.

Basically, all your assumptions are correct: RAID 50 is a striping (RAID 0) of RAID 5 arrays, while RAID 10 is a striping of RAID 1 arrays.

How this is physically implemented, however, depends strongly on the disk controller; sometimes, additional space is used for internal informations, so you can't know exactly how, when and where every single byte is used, unless you ask the controller vendor.

About the stripe size: this is almost never relevant, unless you are into heavy performance tuning; in this case, it can have an impact, but it depends (again) on the controller and disks you are using, and also on the OS, the filesystem and the actual I/O load.

As a rule of thumb, it's good practice to have the stripe size of the RAID array match the cluster size of the filesystem with which the volume residing on that array will be formatted; and that size should be chosen depending on the I/O load the volume is expected to handle (many small files or lots of big files?); but this is only a general suggestion; and, again, lots of other parameters can influence I/O performance.

Also, keep in mind that you could have multiple volumes on the same RAID array (even more so if you are working with a SAN instead of local storage), each of them potentially using a different cluster size and handling a different I/O load.

If you really want to fine-tune your storage to such a level, not only you will need complete control of each and every element from the physical disks to the actual application storing data on them, but you will also have to analyze them carefully and customize a lot of parameters, of which stripe size is only one of many.


A simple case study: Exchange writes database transaction logs as 1-MB files, sequentially; they are mostly written and rarely read under normal operation; they can take up some space, but never too much if regular backups are performed, because they get truncated (i.e. the oldest ones are deleted) everytime a full backup of the database is completed.

The best possible approach for storing this kind of data would be to use a RAID 1 array of two disks, with a stripe size of 1 MB, battery-backed-up write cache, a single volume formatted with the NTFS filesystem and 1-MB cluster size; oh, and of course you'll have to store only the transaction logs for a single database on this volume; if you have more DBs, you will need to use different volumes and disk arrays, or you'll lose all benefits of sequential I/O. (BTW, the actual database data must go to a whole different place, and not only for performance but mostly for data safety; have a look at the Exchange documentation if you want more details; but the basic points are, they have completely different I/O patterns and you absolutely don't want to lose both the database and the transaction logs at the same time.)

As you see, this kind of assessment is very strongly dependent on the expected I/O load, and wouldn't be adequate for anything else than storing Exchange transaction logs in a very specific setup; it will probably hinder any other workload.

Storage fine-tuning is an art, and requires lot of analysis and experience to get it correct.

  • 1
    Well, assuming the controller is using all the disk space and not wasting even a single byte for each sector (and this is a big assumption), your stripe size should be such that each operation on a stripe impacts the same number of sectors on each disk of the array; this is easy to compute when using RAID 1 or RAID 10, but a lot harder with raid 5, 6, 50 or 60. – Massimo May 20 '16 at 21:04
  • 1
    Anyway, if you are really concerned about this, ask your storage vendor; they are the only people with enough low-level information to properly answer your questions. – Massimo May 20 '16 at 21:18
  • 1
    Right. This is boring. Without specifics, it's hard for anyone to give a relevant answer... and again, if performance is what you're seeking, you can't exclude the impact of caching nor ignore the fact that SSDs are the go-to solution for many I/O profiles now. – ewwhite May 20 '16 at 22:36
  • 2
    I'd actually accept any details about how stripes with parity are stored and tracked. To me, it's interesting that no one else is curious about the details. – Max Vernon May 21 '16 at 4:27
  • 4
    @MaxVernon the stripe size is the size of a stripe on a single disk, rather than the whole stripe across all disks. Physical layout on the disk is a bit of a red herring - the disk has different numbers of sectors in different zones and does a translation behind the scenes. Basically if you think in terms of one stripe per revolution of the disk you'll be close enough to optimal that you won't significantly improve on it. – ConcernedOfTunbridgeWells May 24 '16 at 16:03
1

Massimo gave a pretty good summary and as he says, a lot depends on the kind of workload you are running.

In addition, the controllers and their firmware play a big role. For instance, at home I have an LSI 8 port SAS/SATA HBA that can be flashed to run as a RAID controller. The same hardware is badged by Dell but the firmware sets up a different queue depth to support specific Dell disks. My OEM firmware outperforms the Dell firmware by about 30% when using 5x 4TB WD consumer disks in my home machine. If I flash the Dell card with OEM firmware the performance is identical.

ConcernedOfTunbridgeWells notes that you have spinning disk...

If you are able to run this workload on Linux/Unix you might consider one of the filesystems that allows SSD caching of the magnetic disks.

At home I run ZFS on Linux and its extremely reliable due to its flexible parity and continuous hash based consistency checking. It natively supports SSD caching and its lightning fast with only a modest SSD cache drive. The ZFS array with LSI in HBA mode is faster than using the LSI as a hardware array. The workload is Openstack virtualisation (its my lab machine).

Better still is just use a proper SAN or even NAS that knows how to adapt the controllers, caching, striping etc for specific workloads.

  • Thanks for taking the time to answer. I am using a PCIe SSD for the O/S drive - it's blazing fast. However, I'm not interested in finding ways to speed up my RAID array, as much as I'm interested in knowing what impact of specific physical characteristics of the array would be. – Max Vernon May 25 '16 at 17:24
  • 2
    Understood. There are a lot of variables so it would be good to understand what you are trying to optimise for. SSD is easy... sector access is done in constant(ish) time so main factor is how much IO can be done in parallel, how much parity computation has to be done and what percentage of disks are used for parity. Magnetic is different... seek time is important for random access, contiguous sectors per revolution are important for sequential IO. – Tricky May 26 '16 at 22:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.