Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Any suggestions how I can distribute millions of files from 1 server to X number of other servers? I'm looking more into an algorithm on how to decide which server to send the file to.


  • no db
  • perl/python/shell based
  • ability to run form any box and end up on the same destination server

locked by HopelessN00b Jan 21 at 22:23

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

closed as off-topic by HopelessN00b Jan 21 at 22:23

This question appears to be off-topic. The users who voted to close gave this specific reason:

If this question can be reworded to fit the rules in the help center, please edit the question.

Have you looked into distributed filesystems? –  ℝaphink Dec 28 '09 at 8:52
A few people have tuned in with the distributed filesystem, but this doesn't sound like what you want. Can you expand & explain further? –  Philip Reynolds Dec 28 '09 at 11:08
How big is the collection of these million files? git might work, or rsync. –  ptman Dec 28 '09 at 15:07
To keep things simple for now, we're staying away from any distributed file system. We have two storage servers with 22TB each, which are nearing capacity(40TB used now). We've ordered additional storage and servers to upgrade and create new storage servers. I'd like to move some of the existing files onto the other servers. git won't work for us. rsync is a good transport tool, but I would still need a way to determine which files would go to what servers. –  user30199 Dec 28 '09 at 20:44

4 Answers 4

Perhaps look at a distributed filesystem like GlusterFS. It sounds like it will meet all your requirements and will probably be more reliable than something that you hack up yourself.

+1 for GlusterFS. –  Tom O'Connor Dec 28 '09 at 15:29
To keep things simple for now, we're staying away from any distributed file system. I will look into it but adding/changing the architecture at this point is not an option. –  user30199 Dec 28 '09 at 20:47
So, you want to distribute your files, but you don't want a distributed filesystem. Oh, and redistributing your files == changing your architecture. –  womble Dec 28 '09 at 23:05
More concerned about redistributing files files when add more capacity. Files are already distributed so it is not a change. –  user30199 Jan 26 '10 at 2:08

If you still want to hack it, do a md5sum on each file and then hash the output to your X boxes..

If you have two boxes:

0*-7* go to box one 8*-f* go to box two...

Or if you have 256 boxes: 00*-0f* go to box one 10*-1f* go to box two.. and so on..

This works best for box counts of powers of two.(2,4,8,16,..)

Keep in mind that shuffling things off is all nice and good, but you'll want to keep an index somewhere if you also need to retrieve this info.

(where did I put foo.txt??)

A flat file pickle (in python) would work, but it won't scale as well as a DB for large amounts of data..

"00-0f go to box one" means you can address up to 16 servers, not 256. –  womble Dec 28 '09 at 23:33
00 to ff.. good catch. –  Joel K Jan 14 '10 at 23:27

Despite your impossible requirements, I'll scribble down my thoughts for other people in the future who aren't so hamstrung, based on my experiences doing this for Github.

Distributing data across a number of locations (be they partitions, machines, data centres) based on a hash is a dangerous undertaking, for two reasons:

  1. You're never going to get a balanced distribution of data based on your hash -- not necessarily because the hash isn't balanced (although that is a factor too) but more because the items you're storing aren't equal size. So you store two items, one 1kB in size the other 1GB in size. Already you're massively unbalanced. Try that a few times and all of a sudden you've got big imbalances.
  2. Once your hash-to-server algorithm is in place, you can't change the number of "buckets" (machines, partitions, whatever) to store your data in without a massive amount of pain. This is because the hash algorithm is used both to decide where to store stuff, and also where to find it again. If you change the number of servers, then the rule of "where stuff is" changes, and so some of the existing data is expected to be somewhere else. You end up either having a lengthy offline "rebalance" operation (each server searches for data that, in the new scheme, should be somewhere else, and moves it there) or you have to search for your data on all the fileservers (mmmm, inefficient).

On the other hand, having a lookup table for all your files makes these problems go away. When you say "no database", I'm betting you insert an implicit "SQL" before "database". However, there is a whole other world of databases out there that have nothing to do with SQL, and they are perfect for this situation. They're known as "key-value stores", and if you're dead keen on going ahead with building this boondoggle yourself, then I'd highly recommend using one (I've got experience with Redis, but they all seem pretty reasonable).

Ultimately, though, if you go ahead with the "all hashes, all the time" system and then hack around the problems inherent in it (there are solutions, just not real awesome ones) all you will end up with, at the end of the day, is a half-assed, botchy, non-feature-complete version of GlusterFS. If you need a large amount of storage, growable over time, distributed across multiple physical machines, in a single namespace, I really would recommend it over anything you can build yourself.


Can the other servers also send files? Are you in a "safe" environment?

The Rocks clusters installation process has to fill rack after rack of compute nodes, each one installed on the fly from an initial image. Doing that linearly or through a single server would be a bottleneck. Rocks uses instead a little system called Avalanche, where the install images are served using p2p; as nodes come up, they also become servers that will be used to install new nodes. The result is a tree of servers, and the install images cascade through the racks very quickly. The overall latency is a logarithm of the number of nodes, multiplied by the time to install one node (the base for the logarithm depends on how many other nodes can be served from one that is already installed, log base 20 wouldn't be surprising...).

You could imagine a similar strategy for copying out your files, but only if the destination servers would be willing to trust other servers for their copy.


Not the answer you're looking for? Browse other questions tagged or ask your own question.