High availability NFS

Question

We're considering hosting a web application on Amazon AWS. I've drawn a proposed setup for it, which I will try to summarize in a few lines:

• The web application is served from webservers in 3 Availability Zones behind a loadbalancer. The webservers are automatically scaled up from 3 when the load increases. This is done through a user data file that runs a few bash commands.
• The database is placed on a Multi-AZ RDS solution
• Because the application writes to the file system, we also need to mount some kind of network attached file system on the webroot.

The last bit is what worries me. I have some experience with AWS and besides dealing with the latency between two Availability Zones, this would provide a single point of failure.

So, I've been looking at GlusterFS, because that's what someone on serverfault suggested to someone who was dealing with a similar pickle. I've been setting up an environment with a Gluster node in every AZ. In the startup script for my webservers, I evaluate the name of the AZ it's in and pick the Gluster node that's in the same AZ, to reduce latency. That's perfect!

But let's say the node in AZ us-east-1a fails somehow. Is there a way to get my webservers in us-east-1a to fall back to the node in us-east-1b if the node in us-east-1a is unavailable? And, of course, if both are unavailable, also to us-east-1c?

So far, I've only seen examples of people using the server and client functionality of Gluster on the same machines, which I'd like to avoid. It's probably good to note that I will be using the NFS client for performance reasons.

Of course, any other suggestion for this file storage system would be very welcome.

Answer 1

I'd like to update anyone interested on how I decided to go about this. Like I said, S3 doesn't suit our situation which is probably something more people struggle with.

Gluster seemed the way to go at the time, because it's specifically meant to do these things. However, in our testing environment, Glusters speed held us back. Yes, file transfers to a Gluster filesystem are pretty fast, but we do a lot of lookups and quick reads and writes on these mounted volumes and when testing for higher load, it became a bottleneck. The Gluster NFS client is much faster for these operations, but it doesn't support Glusters built-in fault tolerance.

So, we went back to our original idea: Simply using an NFS server with a failover in another AZ. To keep the nodes in sync, we use DRBD. We got around the problem of not having a virtual IP using VTun (I'm very open to other suggestions) and are using Heartbeat to promote the slave when the master is down.