Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I will have 3-4 servers that will need SAN access for both IOPS performance, and raw storage capacity. The limit is, cost. Solutions I have seen run $10,000+ for anything 9TB or higher. If I were to build my own, what hardware would you recommend and what software should be used if it is a $5000-7000 spending limit?

It will all be linux environments.

share|improve this question
Is 9TB the storage target you're trying to reach? – Kevin Kuphal Jun 19 '09 at 17:42
Good question, you're asking a lot though. Storage really does go: Fast, cheap, reliable You can only pick two :-) – nixgeek Jul 4 '09 at 15:49
Will you be using Fibre Channel or Gigabit Ethernet to connect to the SAN? – Joseph Jul 4 '09 at 22:08

10 Answers 10

Openfiler can provide the iSCSI infrastructure for the SAN. Your cost then is simply the disks and enclosure.

You might look at something like this:

which provides 20 hot-swap drive bays for SAS drives. Slap in a motherboard of your choosing and some good SAS controllers. Let openfiler handle the RAID-6 and the iSCSI for you. I think you could do it quite cost effectively.

share|improve this answer
OpenFiler is pretty solid once up and running, but it can be a PITA to configure for the first time. It will probably be worth purchasing the Administrators guide and potentially a support package. – Bob Jun 19 '09 at 20:05
NICE - like that a lot, we always get to buy nice HP servers but it would be so fun to get to build something like this from scratch. – Chopper3 Jun 29 '09 at 18:50

Don't have any hardware recommendations off the top of my head, but as for software and networking you should definitely go for iSCSI. It will allow you to run the SAN network over your existing ethernet infrastructure, and adding more clients is basically free if you use the open-iscsi initiator.

Of course, having a dedicated switch or at least a VLAN is recommended, as well as a dedicated port on each machine for iSCSI traffic. You can get by without this but performance will suffer.

For the target, you can run OpenFiler, which will basically give you an appliance-like solution and ease the burden of configuration.

share|improve this answer
I'm liking what I'm seeing at the OpenFiler web site! – Wesley Jun 19 '09 at 18:30

To really get an answer to your question you'll need to give the community a little more information. Not only the physical specs that you require but also what features are required.

Physical Specs
Interconnect type (Ethernet, FC, etc and what speed and how many ports)
Capacity Required
Form Factor (does it need to be rack mount or stand alone)

What sort of redundancy do you need? Power supplies, controllers, Multi-Path IO
Do you need snapshots?
Do you need replication?
How are you planning on backing it up?

With that said even the entry level Enterprise single controller iSCSI SAN solutions you're going to be looking at $10k+ for 2-5TB of storage. That's just the where the market is. If you're running 4 servers off this one SAN just remember that is a single point of failure that could take down all the services that those servers are running. So you need to plan accordingly.

My favorite storage platform for do it yourself these days is OpenSolaris. With ZFS you get a lot of features only found on the commercial SAN solutions. There's also Nexenta which is commercially supported storage platform based off OpenSolaris and ZFS. It may be a good compromise.

share|improve this answer

EDIT: After another year or two of experience with the Promise, I can safely say - AVOID THIS PRODUCT! Under any amount of heavy load the Vtrak would timeout on certain iSCSI commands forcing us to limit its usage. The last nail in the coffin was when the Vtrak failed 1/2 of the drives after a reboot resulting in data loss.

We put a Promise Vtrak iSCSI SAN in place a couple years ago for some cheap storage and it has worked great for us. You can use off the shelf SATA drives (I recommend Seagate Barracuda ES2) which cuts down on your cost. It will support up to 16 1TB drives depending on the model.

Don't forget to include a dedicated switch (or two) with jumbo frames support and some dedicated NICs for your server. Two NICs and two switches will give you some added redundancy.

share|improve this answer
I have a vTrak 15200 with iSCSI. It is without doubt the biggest piece of junk I have ever bought. Yes, it was cheap - but support and firmware releases died shortly after purchase. It cant create volumes (only arrays), the builtin iSCSI software craps out and send garbled data when trying to export arrays larger than 2TB. The list goes on and on and on. Please - do not buy a Promise array. God will kill a kitten if you buy one. – pauska Jun 20 '09 at 1:32
I have to agree, Promise VTrak are crap. We had two of them, one had heavy load issue just like Doug. The other wiped all the Lun's after a firmware upgrade... – Alex Mar 8 '11 at 16:40
A client of ours had a couple of Promise boxes, these were SAS connected with redundant controllers, one the head unit and the other a JBOD (their "enterprise" hardware), and I agree that they were junk. We weren't even running them that hard, and they would periodically generate I/O errors as one or both controllers rebooted. Firmware that "fixes this problem" was always right around the corner, but new versions never resolved the problem... – Sean Reifschneider Sep 26 '11 at 7:55

You can pick up some very nice, but older SAN solutions on eBay. A friend of mine was looking at an EMC CX500 with like 70 disks for $11k. I'm sure you can find a smaller unit from EMC or another vendor in your price range.

share|improve this answer
Exactly what I was going to say. You can also get recertified SAN equipment that qualifies for vendor support. $5000 will certainly get you a low-mid range recertified clariion. – ConcernedOfTunbridgeWells Jun 29 '09 at 18:46
Don't forget operational costs, not just capital acquisition, something with 70+ drives is going to have very poor performance:watt ratios compared with something newer, costing you lots more to run! – nixgeek Jul 4 '09 at 15:47
This is true, however a SAN with less disks may not be able to handle the IO requirements (which weren't stated, but implied). Monthly power bills are usually easier to get approved than up front purchase costs. – mrdenny Jul 6 '09 at 22:18
Particularly now, a SAN with lots of 15K discs will probably not be able to keep up with just one SSD in terms of IO/s for heavily random I/O. We had a client with 8 10K RPM drives in a RAID-10 array that couldn't keep up and replaced it with a single Intel 600GB SSD and their performance problems disappeared. (It was DRBD replicated to another host, and once we demonstrated it worked they ended up replacing all 8 drives with 160GB SSDs) I can assure you the SSD used less power than the 8 10K RPM drives. :-) – Sean Reifschneider Sep 26 '11 at 8:00

Many people have been excited about the DroboPro. Take a look. It supports everything you want at the price point you provided.

share|improve this answer
My issue with the DroboPro would be the use of SATA drives which would not provide very good IOPs performance for the SAN – Kevin Kuphal Jun 19 '09 at 18:25
Fast, cheap, and reliable. Pick 2. If you want cheap, you're going to need to compromise on speed. – Brian Jun 19 '09 at 19:08
Let's also face it, the number of IOPs provided by a 15K SAS disk over a 7.2K SATA disk isn't double. Not by a long shot ;) Probably the primary reason for investing in 15K is the lower seek times, higher throughput and much higher reliability. If you absolutely must have ultimate performance, then look to some of the HP EVA solutions which now bundle SSDs into the mixture, or perhaps the FusionIO products which are delivering >100,000 IOPs per card. – nixgeek Jul 4 '09 at 15:48

All-Linux environment? All the servers in one place, or at least attached to a single switch? Don't use iSCSI, Use ATA-over-Ethernet (AoE) instead. Similar concept to iSCSI, just cuts out a networking layer to let you use more of your bandwidth. Spend your money on a bunch of harddrives, a case to put them in, extra networking ports, a decent hardware RAID card and a switch capable of Flow Control (a low-end "web managed" switch should be capable of this).

Alternatively, cut the idea of having a SAN at all and keep storage local to each machine. That way you get the 3GB/s performance of a snazzy fibre-channel SAN for the cost of a motherboard capable of SATA II. Buy some extra network cards ($10 each?) and have a dedicated link between each machine, then mirror volumes over those network links between machines.

share|improve this answer

I'd recommend looking at a nexenta/pogo combo. Super robust platform that nicely fits your budget.

share|improve this answer

Well, this is a tough one. You want both high IOPS and high raw storage capacity. Thats kind of the best of both worlds, wich costs money.

I think you have to settle on one of the goals and then try to find a product that suits.

Personally I can recommend Nexsan products. They have rubbish names for their products, their manuals and GUI look like it was made in 1995 but their products are EXTREMELY stable.

We have one 42TB SATABeast with dual controllers (FC and iSCSI), zero problems so far (1 year old). We also have a ATABoy, zero problems except disk dying of age.

They just released their SAS storage arrays. Perhaps a SASBoy will fit your needs? IIRC, you can mix SAS and SATA in the same case (high I/O on SAS, high capacity on SATA).

share|improve this answer
Misread your budget, I thought you said $50.00 and not $5.000. My suggestion about a Nexsan array is not very helpful then, but take my word on this: You will not get high IOPS and raw capacity for $5000. – pauska Jun 20 '09 at 1:44

I am a big fan of the NetGear products. Although they are SATA-2 drives they provide pretty cheap performance. The newly-released 3200 offers up to 12 SATA drives and their proprietary XRAID-2 is pretty fast and convenient.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.