I am trying to configure replicated storage over three sites - each site having a full replica of a file system, using either GlusterFS, or XtreemeFS. This replica must be readable - very fast - locally to each site, without having to access any remote nodes (we intend to use the storage cluster to store a huge number of third party header files, and libraries for a large C++ project). Writes must be possible from any site, but performance is not so important. Synchronous writes would be acceptable. Unfortunately, I seem to be running in to an issue, with how to configure these file systems to select a replica that is local to a site for high performance reads. I have also looked at DRBD, and creating a ring topology, or dual master replication, but rejected this idea, because it just seems too complex. I am software engineer, not IT person, so please forgive any ignorance in this area. I appreciate this this is a very technically difficult problem, and would like a good solution that handles the locking semantics for writes correctly. For Gluster, I can't see any practical way to achieve this. For XtreemeFS, I was assuming that that I could assign a 'datacenter map', and use storage policy WqRq, but unfortunately, this doesn't seem to work. It seems that if I have three replicas, I must have three replicas available to any server that attempts to write (for Gluster, it also seems that all three Bricks will also be written synchronously, regardless of site...). Do I need to set up replicated meta data server for XtreemeFS, each with a different ordering of data center map (to prioritise local replicate) to achieve this? How do I force usage of the local replicat on each site? Mounting appears to require the URL of the directory server, so there does not seem to be any easy way to force a specific ordering of replica selection. Is there any alternative that will allow me to have mutli-master writes, and suitable choice of local replicas, for good read performance?
2 Answers
What you need is a really xpensive SAN... We got in my las work a bunch of ssd acting as cache wich they got accessed the majority of time. And backward the real storage with a bunch of SATA disks... We used it to have file storage + vdi + like 200 vm's on esxi.And all of this storage was going in fibre channels to the other baremetal. http://searchstorage.techtarget.com/tip/Clustering-in-a-SAN-environment-for-high-availability
Master/Master without needing to access remote sites, while still requiring data integrity?
I'm pretty sure that's impossible, if I am understanding you correctly.
You can set up master/slaves (no multi-master), or you can leave some time for replication delay using multiple pools (access to remote sites, again, not master-master, and no guaranteed state across the pool at any given time). Again, a number of solutions to fit your use case, but none of them will have all those requirements.
A master/master filesystem without immediate connectivity to other nodes simply isn't doable. You can't have a single, consistent filesystem if the filesystem can't speak to it's storage easily.
