Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a custom app that is multi-threaded; each thread runs on its own logical core (the workstation is a dual xeon, with 12 physical and 24 logical cores). So there are 24 threads running simultaneously.

I've been researching the multitude of storage options over the last 2 days, and my head is spinning at about 15k rpm.

The app has 2 modes, and they are exclusive: read data or write data; by this I mean that they won't be doing reads/writes interleaved. Each thread will just be doing long sequential reads or writes. The total storage I will need is huge: over 50 tbs (if you are reading this in the year 2016 you are probably having a good chuckle right now over the word "huge".)

Each thread will be reading or writing a file that is about .8tb

I am going to go with jbod, because if a drive fails, all I need to do is swap it, and the app will re-create the data in about 10 minutes.

I will be putting the drives into an external tower or rack, using a SATA OR SAS controller (haven't figured out the +/- of those yet).

So, my question: am I correct in assuming that using 1tb drives for this particular app would be better performance wise than using drives 2, 3 or 4 times that size? It would seem that, unless a 3 tb drive has sequential read/write throughput that is 3x that of a 1 tb drive, the smaller drive is the way to go.

Obviously, using 3 tb drives reduces the number of drives I need to worry about by 1/3, but that would only be a consideration if I could achieve performance that is in the same ballpark as the 1tb drives.

share|improve this question

4 Answers 4

up vote 3 down vote accepted

Most importantly, for best performance you need 24 drives (or more, bear with me), because you have 24 threads. If there are less disks than threads, you do not have sequential operation. Considering two threads on a single disk, it will have some seeks; each seek is say 10 ms, so a loss of about 1 MB of transfer opportunity. With only 10 seeks per second, you have 90 MB/s (2 x 45) instead of 1 x 100 MB/s.

I think you would be better with 48 drives 1 TB each. Drives would be paired, to get 24 stripped (aka RAID0) groups. You can assume that stripping done on OS level will have an unnoticeable impact, so effectively each thread gets double throughput of a 1 TB drive.

I see no possible benefit of 3 TB drives performance-wise.

Still the biggest performance benefit would be something totally different. Just make sure that the application streams the data effectively - that it seriously feeds the queue on the HDD with those I/O commands. If it is written in a way that it clumsily waits for some I/O to get completed before queueing another I/O, then it would severe the throughput.

  • Best to fix it in the application itself or...
  • To partly alleviate this, you would need to invest a lot of money into a disk array with a huge extra cache.

PS. Love the remark about the 2016, hi there folks! Did you get those hover boards already?

share|improve this answer
    
In general I'd agree with your comments about size not relating to performance, however there are some disk models (eg Hitachi 7K3000) that only come in 2TB and 3TB models, which offer significantly greater speeds than the 1TB (Hitachi A7K2000) models. If you believe the datasheets, at least. –  Daniel Lawson Dec 1 '11 at 22:03
    
Excellent point...But why use RAID0? If my 24 threads need to read 1 tb sequentially (and of course simultaneously), are you saying using 12 2tb drives under RAID0 would be just as fast as 24 1tb drives without RAID0? –  user994179 Dec 1 '11 at 22:20
    
Just to clarify my earlier point: the 3TB 7K3000s are definitely not 3x faster than the 1TB A7K2000s. Your approach is definitely better than using 48 single disks. –  Daniel Lawson Dec 1 '11 at 22:27
    
@user994179 I'm saying that each thread should exclusively own a group of two drives working "under RAID0". In your situation this would be much faster than 48 drives in a single large RAID0 group (which is obviously much faster than 12 drives in a single RAID0 group). –  kubanczyk Dec 1 '11 at 23:00
    
I just about put my foot in my mouth by saying use a RAID 10, b/c you wouldn't have any redundancy with a RAID 0 (and then I googled "jbod" at the last minute). Now that I know better, I completely agree with @kubanczyk. I've been doing a lot of research on different RAID configs recently. :) –  David W Dec 1 '11 at 23:44

Well, your answer shouldn't be based on the capacity of the drive, but rather the number of platters and their density. A 7200 RPM drive sequentially reading 100 GB off a 1.5 TB drive composed of 3 500 GB platters will be slower than reading the same 100 GB off a 1.5 TB drive composed of two 750 GB platters because the data density is higher on the latter.

What you actually need to do is find benchmarks for all the drives in question.

I would hazard a guess that in a situation with ~18 3 TB hard drives storing 50 TB vs 50 1 TB hard drives, the 3 TB drives would be faster when reading or writing out one file at a time... but again, it's all down to the individual disks' performance.

share|improve this answer

Considering the operations are sequential and minimal contention, there may not be much of a difference with write performance between SATA and SAS.

Read performance is another story. I don't think you will find many econo SATA 15k drives. The SAS drives tend to be higher end and have the high rpm, and the SAS array controllers usually outperform SATA, but if you are using an econo non-RAID hba, there may not be that much of a difference.

For what it's worth, you should be able to achieve at least 100 Mbytes/second write performance using a single 2TB SATA drive on a standard non-array controller. It could be more with 6 Gbps SATA. For comparison, I have a RAID0 with 4x2TB (6 Gbps) SATA drives, and it has 500 Mbytes/second sequential write performance.

share|improve this answer
    
You can achieve at least 100MBytes/second write performance with a single modern SATA drive only at the outer edge of the platter. The drive spins with a constant rotational speed, which means more bits pass under the heads during a single rotation at the outer edge of the drive than at the innner edge of the drive. As the drive fills up, performance will degrade. 7200 RPM SATA drives can typically only sustain around 60MB/sec at the inner edge of the platter. –  Daniel Lawson Dec 1 '11 at 22:00
    
Are you getting real-world thruput of 500Mbytes/sec? Ie, it will read a 1tb file in ~33 minutes? –  user994179 Dec 1 '11 at 22:14
    
This is a synthetic test, designed to simulate real conditions. But practical usage has confirmed these numbers. The test utility is sqlio.exe, using a 10 Gb file. The sequential test results are typically the least controversial and easily reproduced. –  Greg Askew Dec 1 '11 at 23:34

I think there's some merit in at least trying RAID here. There is some overhead, and RAID rebuilds, when they happen, may end up being a show-stopper, but there are some possible benefits.

  • Much simpler management is the obvious one
  • Single disk performance will peak at something like 50-70MB/sec sustained across the entire drive. It'll be faster at the outer edge (eg, start of the disk), and slower at the inner edge (in my tests, typically half the outer-edge rate) If you're writing to a single disk, you will hit this limit all the time. If you're writing to an aggregate of disks (however it's managed), you'll still hit this limit (sustained performance of a 4-disk RAID0 is simply 4x the inner-edge performance of a single disk), but you'll have higher aggregate throughput.
  • RAID sets will increase contention (if you're doing multiple writes/reads). They will also increase available bandwidth. This may or may not matter to you, but it's worth trying
  • It sounds like your writes are basically streaming bulk data to disk. If that's the case, the RAID5 overhead is fairly low, because if you're writing out full RAID-stripe-width chunks of data to disk, there's no need to do the read/parity recalculation/write cycle - your controller will just write out the new stripe and parity in one hit. (Parity is cheap, it's the read/write cycle that hurts)

Writing out 800GB of data from each thread will take somewhere around 2-3 hours. Assuming your disk will sustain around 120MB/sec at the outer edge, and around 60MB/sec at the inner edge. At 120MB/sec your 800GB of data would take around 111minutes, at 60MB/sec it would be twice that. The write performance drop isn't linear, but it's a fair guess that you'll spend between 2 and 3 hours writing out that 800GB file. @kubanczyk's suggestion probably halves that time, but don't be scared of going to larger aggregations (whether RAID0, or RAID5, or RAID5+0)

I don't have a good feel for this (haven't got round to doing the tests myself), but it's generally considered that SAS drives are better performers, even when the rotational speed is the same (eg, Seagate Constellation ES.2 SATA vs Seagate Constellation ES.2 SAS). Some of the benefits include full-duplex operation for SAS drives and longer queue depth. The full-duplex vs half-duplex point should mean that concurrent read/write operations have less overall effect. You'll still incur drive seeks, but you won't be stalling disk writes while waiting for a read, for example.

I build a lot of systems with 16-disk arrays. Two RAID5 sets (hardware raid controllers), with linux software RAID over the top. We get pretty good performance here: somewhere between 750 and 850MB/sec sustained over the entire array, depending on the disk model used. Throwing in multiple writers does add contention and it does lower the overall throughput, but depending on how often you're writing out your data vs reading it back in, something along these lines may help.

Hope I haven't muddied the waters too much here :)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.