Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm unsure of how to provide a fault-tolerant solution when serving uploaded files.

I have a single load balancer, with two servers (webserver1, webserver2) as nodes, both running Apache2, and a database that is accessible by both nodes, but not the WAN.

The webservers shall eventually be scaled, and the database shall be sharded.

General file replication

At this time, I am running a scheduled rsync from webserver2 to webserver1, webserver1 is the master, who's files are being distributed to webserver2. At this time there are only two servers, though this is intended to scale.

Sharing Images

I have three directories in my Web application that are to be used for user-generated files that are uploaded to both webserver1, and webserver2 (these are exceptions for the rsync operation).

Currently, I have 3 NFS shares being made available from webserver1, said shares are being mounted by webserver2.

I'm quite new to the distributed architecture scene, but from what I can see, the NFS solution leads to a single point-of-failure - I've altered the weighting on the load balancer to ensure that webserver2 receives more traffic, so the load on webserver1 shouldn't be higher.

I've no more budget to spend on any Cloud solutions (such as Amazon S3), so my only course of action is to provide a free software solution.

Are there any viable software solutions? My highest priority is the remove the single point-of-failure whilst keeping server loads to a minimum.

share|improve this question
    
I feel like we're missing information. What type of hardware are you using? Is this a hosted setup? VPS? Cloud? –  ewwhite Feb 7 '13 at 10:15

4 Answers 4

up vote 0 down vote accepted

In this situation, I like to use dedicated NFS servers... Concerns about single-points-of-failure can be offset with certain design decisions...

  • What are you trying to protect against? Component failure? Server failure? Network outages?
  • Where is this setup hosted? A data center facility? An office? The cloud?
  • Will there be a point where budget isn't so constrained?

If the infrastructure is virtualized, I'll have a pair of virtual load balancers (or a single hardware LB), a tier of web servers, DB's (maybe) and backend storage. That storage can be provided by a virtual NFS server, a standalone physical box, a NAS, etc. I rarely use clustered filesystems.

It doesn't seem like you're at the scale where this should be an issue. I understand the motivation to plan for larger scale, but there isn't really anything wrong with what you're doing. Although, an NFS export shouldn't have such a negative impact on server performance either...

share|improve this answer
    
My worry is that the machine will die, or the filesystem becomes unusable - I'm trying to gain as much up-time as possible. The setup is hosted in the cloud, the budget is quite low to be honest but will increase when the load does. After reading your post, it occurs to me that I aught to move my NFS shares onto a volume separate to the system… allowing me to scale the NFS independently. Thank you for your comments (I consider it an answer as you seem to have understood my position and your answer is tailored thus), and others who have posted here too, very good comments, really helpful! –  Nick Feb 7 '13 at 14:28

There's a lot to be said for replicating data between isolated filesystems in terms of performance and resillience. What exactly is wrong with your current set up?

I'd stay away from shared filesystems (GFS2, OCFS etc) They don't give very good performance. A replicating filesystem such as Lustre or AFS offers a reasonable compromise between performance and functionality for this kind of role.

Is there a reason you want to store the data in a filesystem? A NoSQL db cluster might be a lot simpler. And of course, there's also HDFS.

share|improve this answer

DRBD in master/master would cover most of your needs, it doesn't have a high overhead, and either server can write, or read, and will function without the presence of the other.

It is a little complicated to get working, but it is a solution that fits most if not all of your requirements.

share|improve this answer
    
No - DRBD can only be safely used in failover mode - not for concurrent access. –  symcbean Feb 7 '13 at 10:09
    
If you're using OCFS2 or GFS2, that's not true. –  NickW Feb 7 '13 at 10:16
    
That's a recipe for disaster if the cluster goes split-brain. –  symcbean Feb 7 '13 at 10:23
    
I'm not disagreeing there, split brain is painful if it occurs in any situation, in my experience though split brain has only occurred when other things went horribly wrong. –  NickW Feb 7 '13 at 10:33

It sounds like you're looking for something like MogileFS. It's a distributed, replicated file share based on webdav. Mind you, this is not POSIX compliant and your application needs to be 'mogile aware'. It was built to share images for a website.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.