I'm building a 45 drives (backblaze) storage pod for use with Windows 2012 R2 Storage spaces. Spec is 2xE5-2620 v2 cpus, 32 GB ram, 3xLSI 9201-16i cards, 9x1 TB SATA SSDs, and 36x4TB HGST ultrastar SATA drives. Assuming a 50/50 read/write, the raw numbers from the drives clocks in at 840,000 IOPS. The LSI cards can handle 400k+ iops per card. Assuming I put the drives in a 3 way mirror I'll have 48 TB of magnetic storage, using the SSD's as tiered storage. What would be the bottleneck on such a system? Would it be RAM or cpu or LSI card or the drives themselves? Would we see anything close to 50% of that theoretical max in iops?
asked May 14, 2014 at 4:23
-
I'm looking for someone who has real world experience in these types of things to point out if I've overlooked something obvious. More ram, bigger cpus, etc.– chris.w.mcleanMay 14, 2014 at 4:41
-
I understand what you're looking for. All of this is completely dependent on your specific usage and load profile, which is made clear in the linked duplicate question.– EEAAMay 14, 2014 at 4:42
-
@chris.w.mclean What size and type (random/sequential matters a lot for the spinning discs) of IOPS are you looking at? That number seems extremely high. What's the reason for the 3-way mirror? (That will cut your write IOPS to a third of theoretical off the top)– Shane MaddenMay 14, 2014 at 5:01
-
1Geez, I'd be doing this in ZFS. But really, those Backblaze pods are low-quality. I hope this isn't going to run anything critical.– ewwhiteMay 14, 2014 at 5:25
-
The system will be holding a few vms, the workload will be messages dropped into msmq, then passed around to other queues for massaging/correction before being eventually pushed into a sql 2014 db. Messages are between 4k-10MB in size, (average probably around 6k). So I think we will be seeing something close to 50/50 r/w on the disk access.– chris.w.mcleanMay 14, 2014 at 6:08
Add a comment
|
1 Answer
It really depends on what you'll be doing with the system. And this is generic advice; the same for a Windows Storage Spaces solution, versus ZFS, versus traditional filesystems.
Think about it like this:
- You're using a bunch of capacity-optimized disks, so whatever you're doing seems like it involves a large amount of data.
- When people talk about large datasets, the application is usually backup or something where the active working set of data is just a subset of the data on disk.
- If the purpose is backups, why do the specific IOPS matter?
- If the purpose is some other application with a relatively small working set, I'm assuming it can be handled at the SSD caching layer.
- If random read/write performance is important, this is the wrong solution.
- How will you get data onto and off of the server? If anything, that would be your try bottleneck.
- Advanced filesystems luckily don't need much CPU horsepower. Your CPU is what I normally spec in ZFS storage systems. You may want more RAM, but I don't know the specifics of how WSS leverages physical memory.
In reality, your issues will probably be SATA disk timeouts, failed drives, controller issues (firmware), power and cooling, vibration... And bugs. We rarely hear about Windows Storage Spaces here, so my guess is that the industry mindshare is low.
Why can I say this? I've had to administer Backblaze units and personally own a Sun x4540 full of SATA disks (meh). Even with all of those disks, this setup really is only going to be suitable for sequential and streaming workloads.
-
Thanks ewwhite! I think this lets me know that this isn't going to be a solution for what we're planning. Back to the drawing board! May 14, 2014 at 6:09
-
+1 for pointing out the network side. The IO Looks monstrous - but this is a file server. THis is enough IO bandwidth to serve possible multiple 10g connections at the same time (which then hits a memory bandwidth limit).– TomTomMay 14, 2014 at 6:44