We are looking into building an iSCSI SAN for virtualization and general storage needs. At a prior job I had built an iSCSI SAN using opensolaris (back when it was SUN) and utilized snapshot send/recv for sending data to a dr site. This setup is still working well and has not required much attention.

Since the company I work for is growing we are now hitting our limitations and we are now looking at either purchasing a SAN or building it ourselves. At this point, we are wanting three independent nodes, one at our internal site and one at each of our DR locations.

We have narrowed it down to two options as we have attempted to look into the netapp/equallogic/hp msa series side and the cost is ridiculous for what you get.

Option one: Purchase Oracle hardware (a 7410) for each location and utilize the built in replication software (Is this AVS?) to replicate between sites. It looks like it has 12TB RAW which means we may have 4TB after implementing raidz accross the volumes. This is an issue as we will basically be back where we started in terms of maxed out disk space. Yet we now have redundancy. We could add more shelves to get more storage, however at this point we are stuck using Oracle supplied hardware and we are locked in.

Option two: Use Dell hardware, r710s and a dell md1200 arrays at each location. Purchase Solaris through Oracle ($1,000 per processor, so $3k). Where my issue is, is with the replication piece. I used to work with SUN AVS when it was opensource and it was amazing what it could do. I also used to work with snapshot send/recv (inefficient), are you still able to purchase AVS? If so does anyone have any figures on cost of this product? I see on Oracle's website That it is available as a download, however they state you must pay per TB transmitted. Any ideas on cost?

Does the above idea make sense i.e. sanity check? I just want to get the community's opinion and maybe see if others have gone down this path and if there are obstacles I should know about in a larger setup like this.

I have looked at zfsbuild.com and have scoured the internet looking for ideas, my only issue that I cannot find is the replication piece. Thanks in advance for any answers.


I feel like we answer variants of this question often on Server Fault.

You can do all of this with NexentaStor's commercial offering, using either asynchronous replication (ZFS Send/Receive over ssh/netcat or rsync) or synchronous replication (with another commercial plugin).

There are vendors who package and certify these setups with custom hardware. The easiest example is the Pogo Linux Storage Director series.

I personally own a Sun x4540 that used to run OpenSolaris, but is now running Nexenta Enterprise. The Sun hardware is a little sketchy after only two years though, and I've seen found more value in building my own. I'm currently using HP hardware to build single storage nodes, but I can expand to external storage with LSI enclosures. People have documented SuperMicro-based solutions as well. In the HP systems, I'm replacing the Smart Array controllers with LSI 6GB SAS controllers; 9211-8i for internal use, and 9200 for external. You would need to do something similar in the Dell case.

  • Thanks for the information. I know alot of people have asked these types of questions. I asked because I haven't seen any that had asked specifically about replication. I know that nexenta is an option I just don't see what you get for the money. But if that is the only way that I will get replication then so be it. Thanks again! – gdurham Mar 7 '11 at 15:21
  • The paid Nexenta only offers a few things over the free version. One is 18+TB of managed storage. Two, is the GUI built around zfs send/receive and iSCSI LUN mapping. Three, is the availability of the commercial plugins like the Auto CDP and clustering. – ewwhite Mar 7 '11 at 16:31
  • Yeah, I think the only thing that would benefit our environment in this case is the Auto CDP as I have no issue with the command line. Do you feel that zfs send/recv is a viable solution in large environments? – gdurham Mar 7 '11 at 17:11
  • I'm still not certain. I'm going through the process again in a new environment. My main storage needs are for VMDKs, so I'm trying to make zfs send/receive work well for me. I have the bandwidth and low-latency lines, but it's really a matter of how viable/usable the replicated snapshot is on the destination. – ewwhite Mar 7 '11 at 17:27

Option One: out of all proprietary options you'd have, the Oracle ZFS appliances are the ones that lock in the least. You can always drop to the Solaris 5.11 command line and do your send/recv. I didn't understand the 12TB RAW but the way it works is that the 7410/7420 has a head node (or two for clustering) and you add 24-disk shelves to them. We currently have servers with up to 7 shelves but I was told it goes up to 12-14 shelves. It doesn't use AVS but I guess that doesn't matter. It offers scheduled and/or continous remote replication and the iSCSI support is excellent.

Option Two: AVS is no more. You should look at the Oracle Sun Cluster Geographic Edition but I don't think it offers the same functionality that AVS did. Oracle is on the business of selling storage hardware so I hardly think they want people building their own solutions (the 7000s appliances) with their own software (Solaris). They are not going to shoot their own feet like Sun did.

We have a few 7410s with remote replication acting as reliable storage for our Exchange, VMware, Xen and Linux servers with NFS and iSCSI and it works just fine. Take care with OpenSolaris code before the COMSTAR enhancements because in our tests the IOPS was simply to low.

  • With the remote replication, is there a way to do this with standard solaris 10/11 or am I SOL and will need to either do zfs send/recv. What is the average cost of the expansion storage shelves if you don't mind? Thanks! – gdurham Mar 7 '11 at 14:49
  • The remote replication is zfs send/recv. It takes a snapshot and then uses send/rcv to replicate - it just does this very frequently. send/recv isn't guaranteed to be compatible between Solaris/zfs versions though, so you need to bear this in mind with your design. – Adam Mar 8 '11 at 10:40
  • Thanks alot for that comment. I had no idea that this is all it is. – gdurham Mar 8 '11 at 15:57
  • I was told the continuous replication is NOT zfs send/recv because it does synchronous replication and send/recv don't have that capability. Sun acquired some 3rd party code for that. This was told by a pre-sales engineer and I can't directly confirm any of it. – Giovanni Tirloni May 17 '11 at 16:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.