I have a network of 20 machines, running Ubuntu 10.04.

Each machine has about 200[GB] of data that I'd like to share with all other 19 machines for READ ONLY PURPOSES. The reading should be done at the fastest possible way.

A friend told me to look into setting up HTTP / FTP. Is it indeed the optimal way to share data between the machines (better than NFS)? if so, how do I go about it?

Is there a python module that would help in accessing/reading the data?

UPDATE: Just to clarify, all I want is to be able (from within machine X) to access one of machine Ys files and LOAD IT INTO MEMORY. all of the files are of uniform size (500 [KB]). Which method is fastest (SAMBA / NFS / HTTP / FTP)?

migrated from stackoverflow.com Apr 4 '11 at 7:43

This question came from our site for professional and enthusiast programmers.

  • @user540009, "Each machine has about 200[GB] of data that I'd like to share with all other 19 machines for READ ONLY PURPOSES." sounds like you have 200GB/machine. Perhaps you can change your question to reflect only 500KB. – Greg Apr 4 '11 at 4:19

There are hundreds of ways to solve this problem. You can mount a FTP, or HTTP file system over fuse, or even use NFS (why not?). Search for httpfs2 or curlftpfs (or even sshfs, which should not be used if you are looking for performance)

But the problem I see is, you have a single point of failure of the one and only master machine. Why not distribute the storage?

I usually using glusterfs [1], which is fast and can be used in different modes.

[1] http://en.wikipedia.org/wiki/Distributed_file_system


With python you can start up a webserver via a simple one-liner in the directory where the data is stored.

python -m SimpleHTTPServer


It creates a simple web server on port 8000, performance wise I cannot tell you much, and for that type of questions it would be better to ask in SuperUser and not SO.

It doesn't autostart, but it wouldn't be hard to make it.

  • Thank you. Can you elaborate more? how do I access it? does it remain permanent even after the computer reboots? performance vs. NFS? – user76976 Apr 4 '11 at 0:11
  • Updated my response. – mhitza Apr 4 '11 at 0:15
  • WOW, never knew about it! Awesome!!!! +1 – zengr Apr 4 '11 at 0:16
  • @user540009 You can put this command in your .bashrc, so when your machine boots you, the server will startup automatically. – zengr Apr 4 '11 at 0:28
  • python -m SimpleHTTPServer might be suitable as one time thing, no more. – jfs Apr 4 '11 at 0:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.