Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We are looking for a ~32T external storage solution for two 48-core AMD servers. These will be used for a small Linux OpenVZ cloud for CPU intensive web-servers and data warehousing. Dual path with automatic fail-over is pretty much a must. Hopefully the enclosure and the SAS controller would cost around $9k and 16 drives around $4k.

Promise and Infortrend

Has anyone had experience with these systems?

What are some alternatives to the above Promise and Infortrend SAS product for a two server web+db cloud application?

RAID Inc.

This could be a good option: http://www.raidinc.com/xanadu_230.php

SuperMircro

Would something like this also work? https://www.thinkmate.com/System/STX_JE16-0300/14991

share|improve this question
add comment

8 Answers

We just did this here as a test (not my blog):

http://www.natecarlson.com/2010/05/07/review-supermicros-sc847a-4u-chassis-with-36-drive-bays/

It has worked out perfectly. Using off the shelf SuperMicro pieces, we were able to build a 72TB RAW array for about $8,000 total. In hindsight, we didn't need as many SATA controllers, and we never did buy extra disks for caches, as we aren't using the Sun stuff he was talking about.

share|improve this answer
1  
+1 This is impressive! I am looking for something that is super easy to manage (what is your experience?), preferably something without a commodity operating systems and even without a motherboard. –  Aleksandr Levchuk Nov 16 '10 at 16:42
    
We're using Ubuntu 10.04 server with mdadm configuring the RAIDs, exporting it to the nfs server via nbd-server. On the machine itself we get around 680MB/s write and 1.7G/s read speeds. It goes down hill from there when you add nfs on the front end, but that's a different issue. I'd say in the next year, we're looking at building 4~8 of these things. –  erimar77 Nov 16 '10 at 19:45
add comment

some of the supermicro backplanes (the vertical card just behind the disk bays) are in fact SAS2 switches. you can daisy-chain several of these backplanes with one or two 4-lane SAS cables and plug a lot of drives on each.

there's even a 4U 45-bay box with just a power supply and some of these on both front and back; no motherboard! Check the manual for details.

share|improve this answer
    
Check th 4u box for 72 units ;) –  TomTom Nov 16 '10 at 6:34
    
+1 This could be exactly what we need. I assume: (1) this kind of array would be managed by a PCIe RAID controller; (2) if we use 2 4-lane SAS cables then we have redundant path and automatic failover will be possible –  Aleksandr Levchuk Nov 16 '10 at 16:47
    
By the way, the backplanes are also the ones that light up the red-light on the drives when the fail? –  Aleksandr Levchuk Nov 16 '10 at 16:48
add comment

Supermicro has a bunch of well-respected SAS enclosures. I don't have personal experience with them, but have a few acquaintences that have had very good luck. They have a 16-slot model that sounds like just what you're looking for.

share|improve this answer
    
+1 I did not think to look into that. We are using Supermicro for the servers and they are very reliable. But I need it to be a storage-shelf-only because we have all the computing in separate servers. –  Aleksandr Levchuk Nov 16 '10 at 2:36
add comment

More than a year ago, backblaze explained how they build a 67TB server for ~8k$.

This is an interesting article, as it shows the different options they chose, as using a custom enclosure rather than buying one.

They gave all the schematics, and you can buy an enclosure here.

At the time the article was written, a petabyte was 81k$ (raw disks). Using their solution, that was only 117k$ (compared to that, dell was 820k$).

share|improve this answer
add comment

I've had good luck with the storage and server products from Aberdeen Inc. Price wise they are really hard to beat. And they come with a standard 5 year warranty that they do stand behind. It's a win win. I've been using their products for about 8 years now so I have a long relationship with the company.

share|improve this answer
    
+1 I know a sysadmin who uses Aberdeen. There were some issues with newer drives (LSI has that too) but my fellow sysadmin feels pretty confidant about it. Isn't Infortrend the upstream and Aberdeen re-brands and re-writes all the model numbers? What is their contribution? –  Aleksandr Levchuk Nov 16 '10 at 16:34
    
3dinfluence, what kind of setup to you have? Are redundant paths working well? –  Aleksandr Levchuk Nov 16 '10 at 16:35
    
Aberdeen rebrands a few different things that I know of. Their server chassis AIC chassis with supermicro motherboards and Areca RAID. I believe their new ZSAN line is similar but I'm not sure what they are using as the controller. They have two DAS storage series. The P8 XDAS series are rebranded PetaStor which use Areca controllers. Their SAS XDAS series are rebranded Infortrend. –  3dinfluence Nov 16 '10 at 17:18
    
I don't have any experience with multipath with them as I only have experience with their P8 series which doesn't support it. But at my previous job we used about 20TB-30TB worth of them, this was back when 250GB drives were the max size so it was over half a rack of units. We also sold 100TB or so to other customers. The SAS XDAS line supports multipath and I would expect that it works the same as Infortrend. So any good or bad reports you've heard with Infortrend would probably still apply. –  3dinfluence Nov 16 '10 at 17:22
1  
Another company you may want to look into is RAID Inc. I don't have any experience with them but I know others who use and like their solutions. raidinc.com –  3dinfluence Nov 16 '10 at 17:23
show 1 more comment

Coraid is another solution provider I would throw out there. They make a number of enclosures with super micro chassis that might fit your needs. They also seem to be doubling down on future development after closing $25 million in round B funding.

They were recently selected for an Antartic data collection project that's NASA funded, and they've been named as a storage partner by Avaya.

More information about the company and it's health can be found here.

share|improve this answer
    
It's not immediately clear what "Ethernet SAN" is. I can see that they have some SAS drives but do they use SAS for the interconnect or are they providing an alternative solution? –  Aleksandr Levchuk Nov 26 '10 at 5:56
1  
Instead of a standard protocol like iscsi for storage subsystem connectivity with hosts, they utilize a proprietary protocol called ATA over Ethernet (AoE). AoE support is in the Linux kernel, and they make an HBA for windows systems. Theres some pluses and minuses, overall though its a good product at a good price. We have several of their enclosures, and they've been good to us. –  CurtM Nov 26 '10 at 6:26
    
Thanks for the interesting info! I heard that iscsi has really poor performance. It would be great to see a latency, bandwidth, and IOPS comparison study between AoE and SAS 6Gb/s for various types of workloads. –  Aleksandr Levchuk Nov 27 '10 at 21:00
    
AoE technically doesn't have the TCP/IP overhead of iSCSI, so I've seen tests where it's performed better, but I've also seen test's where it's performed worse (which is a bit strange, given the simplicity). Overall I think there's a few main considerations - 1.) do you need your interconnect protocol to be routable? 2.) How much enterprise support do you need? 3.) do you trust CORAID? If you answer no, not much, and sure - then AoE should probably be looked at, especially if you're looking to get the most bang for your buck (or Euro). –  CurtM Nov 27 '10 at 21:29
add comment
up vote 1 down vote accepted

Many good looking solutions were suggested.

RAID Inc. was suggested by @3dinfluence and we are going with this option.

It's a no-single-point-of-failure 6Gb SAS RAID Storage System with 24TB of 6Gb SAS drives

Dual Redundant 6Gb SAS Hot Swap RAID Controller Modules
(4) 6Gbps Serial Attached SCSI (SAS) Ports per Controller
2GB Cache Module per Controller
Dual Hot Swap Power Supply and Cooling Modules
Hot Swap IO Modules, Battery Units, and Management Interface Modules
12-Bay 6Gb SAS Drive Chassis, 2U Form Factor
(12) 2TB 7,200 RPM Hot Swap 6Gb Nearline SAS Drives
Sustained Performance up to 4GB/s
Pre-Read Parity Validation (To prevent against silent data corruption)
Management Software Package for Linux (LSI's SANtricity)
Scalability to a maximum of (192) SAS drives

We are going to pay around $12k and another $1k for the 2 HBAs.

It will be a turn-key solution and all the tech support and part replacement done through RAID Inc.

Thanks to everyone for the answers!

share|improve this answer
add comment

This is a no-single-point-of-failure system. Newegg's prices would be pretty hard to beat.

Total $11k

Still need to buy the HBAs (about $300 each) and 16 Mux Cards for the SATA drives.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.