Best performance: 1tb vs larger drives in app using only sequential read/writes

Question

I have a custom app that is multi-threaded; each thread runs on its own logical core (the workstation is a dual xeon, with 12 physical and 24 logical cores). So there are 24 threads running simultaneously.

I've been researching the multitude of storage options over the last 2 days, and my head is spinning at about 15k rpm.

The app has 2 modes, and they are exclusive: read data or write data; by this I mean that they won't be doing reads/writes interleaved. Each thread will just be doing long sequential reads or writes. The total storage I will need is huge: over 50 tbs (if you are reading this in the year 2016 you are probably having a good chuckle right now over the word "huge".)

Each thread will be reading or writing a file that is about .8tb

I am going to go with jbod, because if a drive fails, all I need to do is swap it, and the app will re-create the data in about 10 minutes.

I will be putting the drives into an external tower or rack, using a SATA OR SAS controller (haven't figured out the +/- of those yet).

So, my question: am I correct in assuming that using 1tb drives for this particular app would be better performance wise than using drives 2, 3 or 4 times that size? It would seem that, unless a 3 tb drive has sequential read/write throughput that is 3x that of a 1 tb drive, the smaller drive is the way to go.

Obviously, using 3 tb drives reduces the number of drives I need to worry about by 1/3, but that would only be a consideration if I could achieve performance that is in the same ballpark as the 1tb drives.

Answer 1

Most importantly, for best performance you need 24 drives (or more, bear with me), because you have 24 threads. If there are less disks than threads, you do not have sequential operation. Considering two threads on a single disk, it will have some seeks; each seek is say 10 ms, so a loss of about 1 MB of transfer opportunity. With only 10 seeks per second, you have 90 MB/s (2 x 45) instead of 1 x 100 MB/s.

I think you would be better with 48 drives 1 TB each. Drives would be paired, to get 24 stripped (aka RAID0) groups. You can assume that stripping done on OS level will have an unnoticeable impact, so effectively each thread gets double throughput of a 1 TB drive.

I see no possible benefit of 3 TB drives performance-wise.

Still the biggest performance benefit would be something totally different. Just make sure that the application streams the data effectively - that it seriously feeds the queue on the HDD with those I/O commands. If it is written in a way that it clumsily waits for some I/O to get completed before queueing another I/O, then it would severe the throughput.

• Best to fix it in the application itself or...
• To partly alleviate this, you would need to invest a lot of money into a disk array with a huge extra cache.

PS. Love the remark about the 2016, hi there folks! Did you get those hover boards already?

Answer 2

Considering the operations are sequential and minimal contention, there may not be much of a difference with write performance between SATA and SAS.

Read performance is another story. I don't think you will find many econo SATA 15k drives. The SAS drives tend to be higher end and have the high rpm, and the SAS array controllers usually outperform SATA, but if you are using an econo non-RAID hba, there may not be that much of a difference.

For what it's worth, you should be able to achieve at least 100 Mbytes/second write performance using a single 2TB SATA drive on a standard non-array controller. It could be more with 6 Gbps SATA. For comparison, I have a RAID0 with 4x2TB (6 Gbps) SATA drives, and it has 500 Mbytes/second sequential write performance.

Answer 3

Well, your answer shouldn't be based on the capacity of the drive, but rather the number of platters and their density. A 7200 RPM drive sequentially reading 100 GB off a 1.5 TB drive composed of 3 500 GB platters will be slower than reading the same 100 GB off a 1.5 TB drive composed of two 750 GB platters because the data density is higher on the latter.

What you actually need to do is find benchmarks for all the drives in question.

I would hazard a guess that in a situation with ~18 3 TB hard drives storing 50 TB vs 50 1 TB hard drives, the 3 TB drives would be faster when reading or writing out one file at a time... but again, it's all down to the individual disks' performance.

Answer 4

I think there's some merit in at least trying RAID here. There is some overhead, and RAID rebuilds, when they happen, may end up being a show-stopper, but there are some possible benefits.

• Much simpler management is the obvious one
• Single disk performance will peak at something like 50-70MB/sec sustained across the entire drive. It'll be faster at the outer edge (eg, start of the disk), and slower at the inner edge (in my tests, typically half the outer-edge rate) If you're writing to a single disk, you will hit this limit all the time. If you're writing to an aggregate of disks (however it's managed), you'll still hit this limit (sustained performance of a 4-disk RAID0 is simply 4x the inner-edge performance of a single disk), but you'll have higher aggregate throughput.
• RAID sets will increase contention (if you're doing multiple writes/reads). They will also increase available bandwidth. This may or may not matter to you, but it's worth trying
• It sounds like your writes are basically streaming bulk data to disk. If that's the case, the RAID5 overhead is fairly low, because if you're writing out full RAID-stripe-width chunks of data to disk, there's no need to do the read/parity recalculation/write cycle - your controller will just write out the new stripe and parity in one hit. (Parity is cheap, it's the read/write cycle that hurts)

Writing out 800GB of data from each thread will take somewhere around 2-3 hours. Assuming your disk will sustain around 120MB/sec at the outer edge, and around 60MB/sec at the inner edge. At 120MB/sec your 800GB of data would take around 111minutes, at 60MB/sec it would be twice that. The write performance drop isn't linear, but it's a fair guess that you'll spend between 2 and 3 hours writing out that 800GB file. @kubanczyk's suggestion probably halves that time, but don't be scared of going to larger aggregations (whether RAID0, or RAID5, or RAID5+0)

I don't have a good feel for this (haven't got round to doing the tests myself), but it's generally considered that SAS drives are better performers, even when the rotational speed is the same (eg, Seagate Constellation ES.2 SATA vs Seagate Constellation ES.2 SAS). Some of the benefits include full-duplex operation for SAS drives and longer queue depth. The full-duplex vs half-duplex point should mean that concurrent read/write operations have less overall effect. You'll still incur drive seeks, but you won't be stalling disk writes while waiting for a read, for example.

I build a lot of systems with 16-disk arrays. Two RAID5 sets (hardware raid controllers), with linux software RAID over the top. We get pretty good performance here: somewhere between 750 and 850MB/sec sustained over the entire array, depending on the disk model used. Throwing in multiple writers does add contention and it does lower the overall throughput, but depending on how often you're writing out your data vs reading it back in, something along these lines may help.

Hope I haven't muddied the waters too much here :)