0

Looking for a windows based solution to create a high performance, 8 GB/s storage array.

Either by striping together dozens of SSDs, or by create multiple RAID arrays and somehow pooling them together. The trick seems to be getting the 8 GB/s from a single volume.... not multiple volumes that each do 1.5 GB/s for instance.

EDIT:

Let's assume read, and let's assume real world scenario (not just theoretical max).

closed as not constructive by Michael Hampton, Jenny D, Jim B, Dennis Kaarsemaker, Dave M May 6 '13 at 17:53

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance. If this question can be reworded to fit the rules in the help center, please edit the question.

9

This is just my kind of question!

Actually creating a single volume that's theoretically capable of this kind of performance on a single server isn't that hard - you need a number of PCIe-based flash drives such as the FusionIO ones I use myself - bound together using LVM or similar (ZFS for instance). This is of course based on a zero-caching model - if you can do it all from memory then life gets suddenly easier but you don't mention volumes so let's assume the worst.

That gets you the actual volume up and running but the issue isn't just that, it's with end-to-end performance.

How are you planning to transit this data? that's ~80Gbps, so you'll need at least a single 100Gbps NIC on a dedicated PCIe 3.0 x16 bus just to clear that amount, plus the same on the storage bus/es of course. Now this CAN be achieved using the right chipset but you're not talking low-spec/cost servers here, I use HP DL980's for this sort of thing and they're not cheap.

Then you come onto the biggest issue, interrupts - there are lots of pretty clever ways of dealing with the interrupt-storm you're creating here but even if you manage to recompile your kernels (Windows, even 2012, will struggle with all this) you're really pushing the actual deliverable, consistent capacity of these servers today - you're asking so much of just about every part of the chain and the weakest part if the most complex, the OS. You also have to understand your write profile as that'll really get in the way of your read performance if it's even remotely heavy.

If you let us know what you're actually trying to achieve here (as this smells a lot like homework to be honest) then we may have some more creative ways of helping but I'd say this is a ~$100k/~2-month project to get going. It can be done (I have lots of servers doing about half this performance day in day out), but you need to know that this isn't a cheap or fast project.

  • everything you say there makes perfect sense. And yeah, it may require a cluster, it will require multiple 10/40 GB ports. I can see how this is remotely possible in Linux, but how would this be done in Windows. – Drob May 3 '13 at 17:06
  • Why don't you tell us the overall objective here, I think we'd be able to help you better if we understood the bigger picture. – Chopper3 May 3 '13 at 17:10
  • What more info would help? – Drob May 3 '13 at 19:46
  • I'm not sure how else to put it - what are you trying to actually achieve? nobody builds a server like this for the hell of it, what's the application? what is the client? why do you want a single server/filesystem to do this? – Chopper3 May 3 '13 at 20:25
  • 3
    Yet you seem really reticent to go into any degree of detail - please read our FAQ about what we're here for and what we expect from those asking questions to show they've done what they can on their own first. – Chopper3 May 4 '13 at 17:21
1

First off, you don't mention if this is read or write bandwidth.

For speeds like that, you're probably looking at multiple PCIe-based SSD cards, and striping them together into a single volume. For example, three Fusion IO ioDrive2's will give you that bandwidth plus a bit extra.

Going with traditional SAS RAID cards is doable but tricky. Those are 6Gb/s per channel, and you'll need 11 channels to reach the speeds you're looking for. There are 4-port cards out there, so you'll need three, and enough SSDs to keep them fed. You'll have to use a mix of hardware and OS striping to make the volume, though.

  • Reality check: SAS has 4 channels per cable. A 4 port SAS card thus uses 16 channels already. As such -1 for bad basics and wrong information. It would be silly to put 255 discs on the bandwidth of a single SATA channel. – TomTom May 3 '13 at 15:02
1

There are a few things to consider here:

  • Capacity - storage that fast is going to be expensive to have a significant amount of capacity
  • Local storage - If this is local storage, PCIe 2.0 is about 500MB/s per lane, which means you'd need something that interfaces with an PCIe 2.0 x16 slot at minimum, or a PCI3 3.0 x8 as 3.0 has twice the bandwidth. The io-drive octal strats to get close but it's only 6.7GB/s read, 3.9 GB/s write for the 10.24TB model, which is massively, massively expensive. You'd have to stripe them together.
  • External storage, If you're talking about non-local storage, the in storage terms the maximum common rate for a single array is going to be only 16Gb/s (2GB/s), so you'd need to multiplex it somehow. I've worked with 20Gb/s fiber, but that's still not going to get you where you want to go. Basically what this means is you just get a lot of fiber, and multiple external storage cabs, and stripe them.
  • Read vs. Write - this will make a big difference in the performance of any solution.

As a shopping question this is probably off topic anyway; as you can see, there are a lot of variables that depend on your specific situation.

  • I think where I am struggling is the "multiplex it somehow" part. I can get multiple RAID controllers in a box that will each do 2 GB/s, but how do I stripe that performance? Just doing it using Windows dynamic disks is a big fail.... – Drob May 3 '13 at 15:12
  • If that's the case then you may need another intermediate layer; what sort of budget are you looking at, or is cost no object? – phoebus May 3 '13 at 15:22
  • Let's ignore cost for now :) – Drob May 3 '13 at 15:23
0

And what is the problem? Except you trying to read 80 gibgabit net data.

  • Use SAS - multiple controllers, external cabinets.
  • Add Storage pools to that (Server 2012)

Finished.

Th problem is doing something with this data. That makes the exercise quite moot.

  • But how do you aggregate that into one volume. Striping in Windows just doesn't get there. – Drob May 3 '13 at 15:10
  • Reading helps. Storage pools. I would also consider doing that on a RAID card level. – TomTom May 3 '13 at 15:45
  • 2
    I don't know of any single RAID card that can pull 8GBps. You'd need multiple cards; and need to think about PCIe Bus limitations too. – Chris S May 3 '13 at 16:11
  • Not that big a problem, the pci-e side. I would suggest the Adaptec line, then not necessarily use a calculation heave RAID - mirroring does not tax the CPU. You still need multiple cards, as I said. And a 40 lane pci-e setup. That is really taxing hardware - maybe doable but has to be experimented with. – TomTom May 3 '13 at 16:15
  • Have you done any testing striping together two 3 GB/s RAID arrays? I have never gotten it to result in anything but a modest boost over the single array..... – Drob May 3 '13 at 17:08

Not the answer you're looking for? Browse other questions tagged or ask your own question.