0

I apologize in advance if this question may (by necessity) be somewhat broad. I will try my best to keep it in the format of a valid question.

I am currently setting up a backend for serving large amounts of static files (images, to be specific). Overall traffic for the final product is expected to be massive, so scalability and the evasion of all kinds of bottlenecks is a must.

My initial conclusion is that the best approach to doing this is setting up a cluster of virtual servers to handle requests. I will use standard load balancing strategies in order to make sure client requests are distributed equally among nodes in the cluster, so this part is no problem.

The part which is a problem is how to efficiently store the static files themselves on the cluster. We are expecting a very large amount (Terabyte range) of data to be present, and keeping mirrored copies of files on each node is simply not an option.

It would seem that I am essentially left with 2 options:

1) Use a central fileserver which the nodes can make requests to. I find it hard to see how I can stop this server from becoming a bottleneck, however.

2) Use 1) in conjunction with some kind of cache mechanism on the node, in order to avoid redundant requests to the file server.

3) Some other brilliant solution which will save my soul and render 1) and 2) obsolete.

What would be a good strategy to obtain this kind of file distribution, and what software is available to do so?

Thanks a lot in advance!

2
  • 1
    what i've seen is for folders to be broken out by the first 2 characters of the filename. so you have your root then you have folders with names '0', '1', so on up to 9, 'a', 'b' up to f (our files are named with hex) then in each of those folders, you have a full set of the same folders, then the actual files are in those. This way, we don't end up with too many files in a single dir. In your case you could even break them out over servers, not just folders, so your clients would know where to look based on the first chars of the filename requested. Aug 30, 2014 at 12:55
  • 1
    Isn't using a CDN an option? Aug 30, 2014 at 13:00

1 Answer 1

2

A consistent ring hash is a classic solution to this sort of problem, at least algorithmically. Systems like Swift, memcached, Dynamo, Riak, have used this approach to address this problem.

OpenStack Swift specifically is an object/blob store that uses a consistent ring hash for tracking and distribution of objects, containers, etc. You could use Swift or something like it directly to solve for your scenario, or build your own solution using a consistent hash ring. Using a solution like Swift has the benefit of not needed to develop it yourself, plus includes things like data protection via replication, versioning, etc.

If you don't have to run it yourself but instead are interested in hosted services, products like Rackspace Cloud Files (based on Swift), or Amazon S3 provide object storage and retrieval fronted by CDNs (Akamai and CloudFront for Rackspace and Amazon respectively). A CDN would greatly increase the efficiency of retrieval for geographically distributed clients.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .