I'm in need of some fast (15+ GB/s write) and somewhat large (50+ TB) storage subsystem with a standard storage interface. I'd prefer something other than Fibre Channel, which admittedly appears to be the most standard interface. RAID 5 is sufficient.

What should I be looking for when purchasing a storage system of this size? How does one build a system that can handle this sort of throughput.

We'd prefer it to fit into less than 20U and cost under a million US dollars.

  • +1 for having a realistic budget. – Tom O'Connor Feb 22 '11 at 21:03
  • RAID 5 for a large array will take forever to rebuild. RAID6 is the way forward. – Tom O'Connor Feb 22 '11 at 21:04
  • Deleting question as per owner's request as it contained priveliged information – Mark Henderson Mar 24 '11 at 3:49

So you want 15GB/s of throughput, and 50TB of capacity, and you want to do it cheaply? Well, I guess that depends on your definition of cheap.

Have a look at this Cost Per Petabyte (about 1/3 of the way down the page). Now, that's of raw SATA storage. SATA is not fast. It's certainly not 15Gb/sec fast. So feel free to, say, double or triple those costs (before you get excited about backblaze, the IO on those boxes is going to be rubbish. They stream their data out over the internet and probably don't even need to saturate a 1Gb/link for each of their arrays).

All disk arrays of this calibre are 19" rack mountable. All of them. And what is "small"? 3U? 30U?

Your other challenge is getting 15GB/sec of throughput. 15GB/sec is 120Gb/sec, and given that the fastest interface I've seen on a single disk array is 10Gb/sec, you're going to need a minimum of 12 arrays to get that kind of throughput. That's assuming that a single array can saturate a 10Gbps link. At 3U per array you're already up to 36U, or almost a full rack.

This is totally unofficial, but based around some pricing that we recently received:

12x 16 Disk storage array (10Gbe, 16x600Gb 15k SAS drives configured in RAID-10) will roughly get you what you need. That's accounting for redundancy and whatnot. They're 3U and let's say $50,000 each. That's $600,000 and 36U of space. Plus the costs of the rest of the 10Gb infrasturcture, expect to be spending closer to $700,000.

| improve this answer | |
  • Thanks for the response, I updated my question. Where did you see the fastest interface on an array at 10 Gb/s? An array is free to have more than one interface card or connection. e.g. ramsan.com/products/3 – Eruditass Feb 21 '11 at 22:26
  • @Eruditass - the arrays we've looked at generally have 2x or 4x 10Gbps connections, but you have to plan as if only 50% of them are working, in case you lose some of your infrastructure and your MPIO routes all fall back to sharing the same interfaces. Having 40Gb/sec is great (although you can't share all 40Gb to the one destination, but that's another topic), but the day you lose one of your SAN switches and you're trying to squeeze 40Gb of traffic through a 20Gb connection is the day you wish you had more arrays. – Mark Henderson Feb 21 '11 at 22:37
  • @Erud - You can also make the arrays much, much cheaper by using RAID6 (or RAID5) and increasing the usable capacity vastly, but then you're potentially sacrificing speed and availability. But if "cheap" is < $1M, then you'll have absolutally no trouble filling that solution. – Mark Henderson Feb 21 '11 at 22:38
  • great points. I'm new to the storage arena so all this is very helpful in deciding what to ultimately go with. Do you recommend any specific vendors that would be good to look at for building me such a solution? – Eruditass Feb 21 '11 at 23:15
  • 3
    @Erude - the big ones are Dell Equallogic and HP Lefthand, there's also EMC and a multitude of them. For a system of this size, I would budget for at least 80 hours of full-time research, just sitting at the vendor websites, plotting all their details, and then spending a very long time on the phone to each vendor and sitting through fun tech meetings, followed by shoot-me-in-the-face management meetings. – Mark Henderson Feb 21 '11 at 23:34

Consult with a reseller who has solution architects on site and who will come up with a solution for you. If you've never done this before, you want to engage experts.

A good VAR will save you money and make sure your solution fits - they want you to come back again.

Disk throughput is only one component of getting such high throughput to the servers that require it.

| improve this answer | |

Check out Isilon unified scale-out storage.

| improve this answer | |
  • Definitely fast, also very configurable, secure and robust, but not necessarily cheap. – MaQleod Aug 24 '11 at 3:05

You may want to check this out http://www.nexsan.com/sasbeast.php. It is high density solution so you will definitely fit into rack space. It should reach your performance goals too, I believe they support up to 2 dual-port FC cards per enclosure. Pricing is competitive too.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy