Distribute files across X number of servers

Question

Any suggestions how I can distribute millions of files from 1 server to X number of other servers? I'm looking more into an algorithm on how to decide which server to send the file to.

Requirements:

• no db
• perl/python/shell based
• ability to run form any box and end up on the same destination server

Answer 1

Perhaps look at a distributed filesystem like GlusterFS. It sounds like it will meet all your requirements and will probably be more reliable than something that you hack up yourself.

Answer 2

Despite your impossible requirements, I'll scribble down my thoughts for other people in the future who aren't so hamstrung, based on my experiences doing this for Github.

Distributing data across a number of locations (be they partitions, machines, data centres) based on a hash is a dangerous undertaking, for two reasons:

1. You're never going to get a balanced distribution of data based on your hash -- not necessarily because the hash isn't balanced (although that is a factor too) but more because the items you're storing aren't equal size. So you store two items, one 1kB in size the other 1GB in size. Already you're massively unbalanced. Try that a few times and all of a sudden you've got big imbalances.
2. Once your hash-to-server algorithm is in place, you can't change the number of "buckets" (machines, partitions, whatever) to store your data in without a massive amount of pain. This is because the hash algorithm is used both to decide where to store stuff, and also where to find it again. If you change the number of servers, then the rule of "where stuff is" changes, and so some of the existing data is expected to be somewhere else. You end up either having a lengthy offline "rebalance" operation (each server searches for data that, in the new scheme, should be somewhere else, and moves it there) or you have to search for your data on all the fileservers (mmmm, inefficient).

On the other hand, having a lookup table for all your files makes these problems go away. When you say "no database", I'm betting you insert an implicit "SQL" before "database". However, there is a whole other world of databases out there that have nothing to do with SQL, and they are perfect for this situation. They're known as "key-value stores", and if you're dead keen on going ahead with building this boondoggle yourself, then I'd highly recommend using one (I've got experience with Redis, but they all seem pretty reasonable).

Ultimately, though, if you go ahead with the "all hashes, all the time" system and then hack around the problems inherent in it (there are solutions, just not real awesome ones) all you will end up with, at the end of the day, is a half-assed, botchy, non-feature-complete version of GlusterFS. If you need a large amount of storage, growable over time, distributed across multiple physical machines, in a single namespace, I really would recommend it over anything you can build yourself.

Answer 3

If you still want to hack it, do a md5sum on each file and then hash the output to your X boxes..

If you have two boxes:

0*-7* go to box one 8*-f* go to box two...

Or if you have 256 boxes: 00*-0f* go to box one 10*-1f* go to box two.. and so on..

This works best for box counts of powers of two.(2,4,8,16,..)

Keep in mind that shuffling things off is all nice and good, but you'll want to keep an index somewhere if you also need to retrieve this info.

(where did I put foo.txt??)

A flat file pickle (in python) would work, but it won't scale as well as a DB for large amounts of data..

Answer 4

Can the other servers also send files? Are you in a "safe" environment?

The Rocks clusters installation process has to fill rack after rack of compute nodes, each one installed on the fly from an initial image. Doing that linearly or through a single server would be a bottleneck. Rocks uses instead a little system called Avalanche, where the install images are served using p2p; as nodes come up, they also become servers that will be used to install new nodes. The result is a tree of servers, and the install images cascade through the racks very quickly. The overall latency is a logarithm of the number of nodes, multiplied by the time to install one node (the base for the logarithm depends on how many other nodes can be served from one that is already installed, log base 20 wouldn't be surprising...).

You could imagine a similar strategy for copying out your files, but only if the destination servers would be willing to trust other servers for their copy.