1
vote
6answers
404 views

Distributed file system with local disk cache

My server infrastructure is growing fast and I decided to create a distributed storage cluster. I've been looking for a proper filesystem for this task which meet my requirement, but none of them ...
3
votes
1answer
142 views

Choice of distributed filesystem for intensive reads and writes

I have a series of servers (HP ProLiant, 34 servers) each of which with 500 G of hard drive space. These servers are part of a computational cluster which runs processes that roughly fall into two ...
2
votes
1answer
101 views

Fault-tolerant network access to redundant storage in a heterogeneous environment (including Windows)?

We are facing a design "challenge" where we are required to set up a storage solution with the following properties: What we need HA a scalable storage backend offline/disconnected operation on ...
2
votes
0answers
143 views

NFS showing files in directory, but can't open or stat

I'm using a network of Linux (Debian Squeeze on kernel 2.6.32) machines, sharing files using NFS (v3). The scenario is that a process running on client A will create a file through NFS on file server ...
2
votes
1answer
35 views

distributing access across multiple disks

I have a rather large application that requires simultaneous access to rather a lot (300+) of large files, and accesses them by reading them all at the same time. If all the files are put on 1 disk ...
0
votes
1answer
81 views

In search of a distributed file-level filesystem [closed]

I'm looking for a distributed, fault-tolerant file-level (not block-level, preferably) filesystem for use in a small Linux-based HA cluster of sorts. The files that are stored will be fairly small ...
1
vote
1answer
378 views

Distributed filesystem for automated offline data mirroring [closed]

I'd like to achieve the following setup: Every time I connect my laptop to a local network, my partition gets automatically mirrored to a partition on my local server. I only want to mirror what has ...
3
votes
1answer
228 views

Are there any serverless distributed filesystems?

Similar to Is there a Distributed SAN/Storage System out there? or Distributed redundant storage solution?, except with a couple key differences (and two years newer). Context: I have a small Rocks ...
-1
votes
1answer
103 views

Availability of big files on multiple servers

I have to handle many (1'000 - 30'000) big files ranging from 200MB up to 2GB. The demand for these files is variable (0 - 300 downloads / file). This is why a single file must saved on 2 or more ...
5
votes
1answer
120 views

deduplicating and indexing directories of images across 150 linux machines

I have a client with 150 Linux servers spread about various cloud services and physical data-centres. Much of this infrastructure is acquired projects/teams and pre-existing servers/installs. The ...
3
votes
1answer
859 views

How to install ceph on EC2 Amazon Linux AMI

I want to test Ceph (a distributed network storage and file system) on some EC2 hosts which is derived from Amazon Linux AMI (amzn-ami-2011.09.2.x86_64-ebs). The kernel version is 3.2 and btrfs is ...
3
votes
2answers
218 views

Distributed filesystem across a slow link

I have an image in my head where a link is too slow to realize the real-time transfer of files, but fast enough to catch up every day. What I'd like to see is a master <-> master setup where when I ...
1
vote
2answers
1k views

linux distributed file system over WAN advice

I have a fairly simple (not really) requirement but I've looked at a few solutions and can't find a good solution. I've got a Red Hat EL 6 server environment in my co-location and office, and some ...
3
votes
2answers
670 views

Linux filesystem or CDN for millions of files with replication

Please tell me solution for this scenario: several millions files, located in one directory ("img/8898f6152a0ecd7997a68631768fb72e9ac2efe1_1.jpg") ~80k file size in average 90% random read access ...
13
votes
1answer
7k views

Why doesn't SSHFS let me look into a mounted directory?

I use SSHFS to mount a directory on a remote server. There is a user xxx on client and server. UID and GID are identical on both boxes. I use sshfs -o kernel_cache -o auto_cache -o reconnect -o ...
0
votes
1answer
945 views

Is it possible to set the cache size for fuse and/or sshfs?

I can't seem to find something in the documentations. Is it possible to set the cache size for fuse and/or sshfs and if so, how?
2
votes
4answers
728 views

Is Samba the right choice for me?

we have a distributed application which uses large amounts of content (all kind of files). There are several servers that need to access the content. Right now the content is stored on each server ...
3
votes
2answers
404 views

Distributed, Parallel, Fault-tolerant File System with high throughput

I am looking for DFS (distributed file system) that is fault tolerant and easy to maintain. I will have tons (100M+) of small files (from 1K to 500K). Files will be located in some directories what ...
1
vote
2answers
489 views

GlusterFS with RAID-1

Is it possible to use GlusterFS to have more flexible FS and to support higher capacity + RAID-1 to have redundancy to avoid data loss in case of failure? I need to get a mirrored FS that has high ...
6
votes
9answers
2k views

Geographically distributed file system with preferred locality

I'm building a application that needs to distribute a standard file server across a few sites over a WAN. Basically, each site needs to write a lot of misc files of varying size (some in the 100s MB ...
4
votes
5answers
1k views

Setting up distributed fault tolerant storage at home

I am tired of worrying about data loss at home. My wife is a semi pro photographer, and essentially all of our family memories are digital (and we ought to convert the ones that are not). I am ...
0
votes
1answer
191 views

Distributed storage with UNC share?

What approach would you take to build a massive storage, using multiple PC's running Linux OS? The recipe I would like to use sounds like this: take 10 PCs equipped with six 2TB disks, install your ...