Questions tagged [distributed-filesystems]
The distributed-filesystems tag has no usage guidance.
128
questions
1
vote
1
answer
703
views
Storing Large Amount of files in a file system?
Guys I am creating an application which will be saving a .txt file (approx 10 KB/file I guess) in a file system and will be saving its path in MS SQL database. Now as per my estimation, approximately ...
- 119
2
votes
3
answers
3k
views
LizardFS/MooseFS -- Multiple Storage Volumes
I'm using the new LizardFS and it's awesome! I'm new to LizardFS/MooseFS overall, but it seems pretty straight-forward.
I was wondering two things:
First, is there a way to setup separate storage ...
- 407
5
votes
2
answers
2k
views
Is there a way to connect a remote folder with local cache on OSX or Windows workstation?
A company with 50 graphic designers are working on the same file server. They are using mainly Indesign.
A typical project is a 60 pages indesign document with 1.5GB of linked files (something like ...
- 224
5
votes
1
answer
4k
views
Why is GlusterFS so slow here?
We've set up a mirroring pair of GlusterFS servers. No special tuning, whatever came "out of the box" with GlusterFS-3.5.1 in the official RHEL6 RPM, that's what we have.
The cluster works, but the ...
- 2,377
6
votes
0
answers
182
views
Fast distributed filesystem for a large amounts of data with metadata in database [closed]
My project uses several processing machines and one storage machine. Currently storage organized with a MSSQL filetable shared folder. Every file in storage have some metadata in database.
Processing ...
1
vote
2
answers
414
views
Setting up a multi-host SAN
I'm looking for some advice on setting-up multi-host I/O access on our SAN:
I have a blade enclosure (PowerEdge1000e) containing an Equallogic PS-M4110 storage blade with a single RAID6 volume ...
1
vote
2
answers
533
views
Simple, decentralized, automatically synced, local cached file share for linux server
I wonder if someone could recommend me some solution to sharing files between a small amount of linux VPS (virtual private servers).
Basically I have a bunch of linux VPS to manage, and I want to ...
- 229
5
votes
2
answers
3k
views
what distributed file system for a two-node failover setup?
I'm trying to set up a redundant setup consisting of two servers that have everything redundant:
the database (MySQL master-master in active/passive mode)
the file system (distributed/replicated)
...
- 453
4
votes
1
answer
3k
views
In a distributed filesystem like MooseFS or XtreemFS, should individual nodes expose "raw-er" storage, or LVM'd storage?
When preparing an infrastructure to utilize a distributed storage system like MooseFS or XtreemFS, how should individual nodes present storage to the rest of the environment?
Is it better for ...
- 18.7k
2
votes
4
answers
2k
views
Scale-out distributed storage with snapshots
I'm aware there are many similar questions around, several with great answers. I still haven't quite found what I'm looking for though: a distributed, scale-out FS that supports snapshots. Gluster ...
user206444
0
votes
2
answers
85
views
Distribute all services between 3 VPS [closed]
i just rent 3 vps servers with this:
4GB RAM
150GB hard disk
2 CPU
6 ip address
so we have vps1, vps2 and vps3 and i'm thinking
distribute the services like this:
VPS1 - frontend
+ DNS Server
+ ...
- 204
0
votes
3
answers
642
views
File system benchmark that generates I/O from distribution of file sizes [closed]
I am looking for a file system benchmarking tool to test CephFS. This benchmarking tool has to:
Generate a set of files (call it fileset) following a particular distribution, e.g., gamma, empirical, ...
- 9
7
votes
9
answers
7k
views
Distributed file system with local disk cache
My server infrastructure is growing fast and I decided to create a distributed storage cluster. I've been looking for a proper filesystem for this task which meet my requirement, but none of them ...
- 121
1
vote
2
answers
260
views
Distributed storage and computing [closed]
Dear Serverfault community,
After researching a number of distributed file systems for deployment in a production environment with the main purpose of performing both batch and real-time distributed ...
3
votes
3
answers
858
views
Choice of distributed filesystem for intensive reads and writes
I have a series of servers (HP ProLiant, 34 servers) each of which with 500 G of hard drive space. These servers are part of a computational cluster which runs processes that roughly fall into two "...
- 225
2
votes
1
answer
573
views
Fault-tolerant network access to redundant storage in a heterogeneous environment (including Windows)?
We are facing a design "challenge" where we are required to set up a storage solution with the following properties:
What we need
HA
a scalable storage backend
offline/disconnected operation on the ...
- 376
1
vote
3
answers
646
views
Distributed filesystem for storing temporary files
We have a cluster of web servers that currently stores temporary files (that have been uploaded by users) on a central shared filesystem. The limitation being that anytime the server containing the ...
- 707
2
votes
0
answers
967
views
How do I speed up “ls -l” on a network file system (NFS)?
I have a NFS volume mounted in the following way:
theHostname.com:/path/on/theHostname on /local/path type nfs (rw,nosuid,addr=10.0.0.54)
The /local/path directory has 30_000 files.
When I do a ls ...
- 465
3
votes
0
answers
3k
views
NFS showing files in directory, but can't open or stat
I'm using a network of Linux (Debian Squeeze on kernel 2.6.32) machines, sharing files using NFS (v3). The scenario is that a process running on client A will create a file through NFS on file server ...
- 469
2
votes
1
answer
86
views
distributing access across multiple disks
I have a rather large application that requires simultaneous access to rather a lot (300+) of large files, and accesses them by reading them all at the same time.
If all the files are put on 1 disk (...
- 255
0
votes
1
answer
142
views
Migration from ext3 to Mogilefs
I have gigs of images on ext3 file system and I want to move them to Mogilefs on a live running site. In your opinion what would be the best approach?
- 145
0
votes
1
answer
134
views
In search of a distributed file-level filesystem [closed]
I'm looking for a distributed, fault-tolerant file-level (not block-level, preferably) filesystem for use in a small Linux-based HA cluster of sorts. The files that are stored will be fairly small (I'...
- 953
3
votes
0
answers
814
views
Are snapshots and clones filesystem-wide in ZFS-backed Lustre clusters
My goal is to find a distributed filesystem on Linux that supports ZFS-like lightweight snapshots and snapshot clones. This StackOverflow question expresses what I'm looking for pretty well. I'm ...
- 31
4
votes
2
answers
270
views
Distributed storage [closed]
At my University Department we are about to upgrade the computers of our student lab (about 25-30 machines). The machines will be running Linux.
One thing about the new machines is that they have ...
- 141
21
votes
4
answers
37k
views
Are ZFS clustered filesystems possible?
Is it possible to create a ZFS cluster? Or would you need to go the seemingly ugly (to me at least) route of ZFS with UFS formatted pools governed by GlusterFS?
The idea is to see if it is possible ...
- 429
0
votes
1
answer
171
views
SQl Server sharding
Can SQL Server be sharded for distributed data processing? Or are there any limitations that prevent relational databases from being sharded ?
- 111
3
votes
3
answers
1k
views
failsafe RAM drive solution [closed]
I'm looking for a production solution to create a RAM drive that will be safely synchronized with HDD.
I have a piece of custom software with heavy I/O load (this is some kind of proprietary document-...
- 151
1
vote
1
answer
664
views
DFS - Stop sync of large folder that has since been removed
We have a site to site DFSR on Windows Server 2008 R2 that has been running perfectly between site A to site B until someone dumped a 20GB folder. This has overwhelmed the upload and make the internet ...
- 1,150
3
votes
1
answer
1k
views
Distributed filesystem for automated offline data mirroring [closed]
I'd like to achieve the following setup:
Every time I connect my laptop to a local network, my partition gets automatically mirrored to a partition on my local server.
I only want to mirror what has ...
- 635
3
votes
1
answer
367
views
Are there any serverless distributed filesystems?
Similar to Is there a Distributed SAN/Storage System out there? or Distributed redundant storage solution?, except with a couple key differences (and two years newer).
Context: I have a small Rocks ...
- 176
3
votes
2
answers
12k
views
Umount stale glusterfs partition
I am using glusterfs on several Ubuntu servers: two of them are running glusterfs servers in replication mode.
Without any clear error, the glusterfs partition became stale and the system shows this ...
- 36.9k
-1
votes
1
answer
113
views
Availability of big files on multiple servers
I have to handle many (1'000 - 30'000) big files ranging from 200MB up to 2GB. The demand for these files is variable (0 - 300 downloads / file). This is why a single file must saved on 2 or more ...
- 11
5
votes
1
answer
161
views
deduplicating and indexing directories of images across 150 linux machines
I have a client with 150 Linux servers spread about various cloud services and physical data-centres. Much of this infrastructure is acquired projects/teams and pre-existing servers/installs.
The ...
- 11.3k
3
votes
1
answer
158
views
Is distributed storage on an internal network possible?
I've been thinking about all the workstations I have on my shop floor (about 50) and the wasted drive space each of them has. For instance, my machines only use about 30G to 40G of local storage, yet ...
- 465
3
votes
1
answer
3k
views
How to install ceph on EC2 Amazon Linux AMI
I want to test Ceph (a distributed network storage and file system) on some EC2 hosts which is derived from Amazon Linux AMI (amzn-ami-2011.09.2.x86_64-ebs).
The kernel version is 3.2 and btrfs is ...
- 261
0
votes
1
answer
2k
views
GlusterFS - Unable to find hostname error
I have installed GlusterFS 3.2.6 on several servers running on CentOS 6.2. I don't have working hostnames, as it's a test setup, so I have just added the servers' IPs to the /etc/hosts file. The ...
- 1,114
4
votes
2
answers
716
views
Distributed filesystem across a slow link
I have an image in my head where a link is too slow to realize the real-time transfer of files, but fast enough to catch up every day. What I'd like to see is a master <-> master setup where when I ...
- 20.7k
4
votes
2
answers
3k
views
linux distributed file system over WAN advice
I have a fairly simple (not really) requirement but I've looked at a few solutions and can't find a good solution. I've got a Red Hat EL 6 server environment in my co-location and office, and some ...
- 161
5
votes
2
answers
1k
views
HW/SW Design: 2 Petabyte of storage
Disclaimer Yes I'm asking you to design a system for me :)
I'm tasked to design a system to store about 10 TB / day with a retention time of 180 days.
My first approach would be to go with GlusterFS ...
- 6,498
2
votes
1
answer
2k
views
GlusterFS vs Mogile vs Others?
I have been looking at GlusterFS and MogileFS (Ceph and BTRFS are off the cards for me, as they don't seem production ready yet).
I am developing a web app for file sharing, that will store and ...
- 4,242
3
votes
1
answer
2k
views
GlusterFS vs Ceph, which is better for production use in 2012? [closed]
This is the same question that was asked here, but it's been almost two years since.
Meanwhile Ceph has seen constant development (361 kernel commits) and btrfs, in my opinion, is on the verge of ...
- 925
-1
votes
1
answer
70
views
When looking for a good network connection speed, why does "bandwidth" not tell the whole story?
While I was listening in on an interesting conversation about distributed systems, I heard people talk about how overhead and "gap" were two important characteristics of distributed systems, and how ...
- 147
4
votes
2
answers
2k
views
Linux filesystem or CDN for millions of files with replication
Please tell me solution for this scenario:
several millions files, located in one directory ("img/8898f6152a0ecd7997a68631768fb72e9ac2efe1_1.jpg")
~80k file size in average
90% random read access
...
- 95
0
votes
4
answers
610
views
Distributed filesystem - confused
I am new to this concept. :)
Can I use a distributed filesystem in as a way to have redundancy for web applications?
Server A and Server B would have Apache/mySql installed on the distributed ...
- 1,025
1
vote
3
answers
911
views
Moving Away from NFS
I've been working on moving more and more services away from a reliance on NFS, and I'm wondering what others have done with this issue. I know there are distributed file systems out there, and I ...
- 268
37
votes
1
answer
34k
views
Why doesn't SSHFS let me look into a mounted directory?
I use SSHFS to mount a directory on a remote server. There is a user xxx on client and server. UID and GID are identical on both boxes.
I use
sshfs -o kernel_cache -o auto_cache -o reconnect -o ...
- 2,403
1
vote
1
answer
6k
views
Is it possible to set the cache size for fuse and/or sshfs?
I can't seem to find something in the documentations. Is it possible to set the cache size for fuse and/or sshfs and if so, how?
- 2,403
2
votes
4
answers
2k
views
Is Samba the right choice for me?
we have a distributed application which uses large amounts of content (all kind of files). There are several servers that need to access the content. Right now the content is stored on each server ...
- 2,403
3
votes
1
answer
2k
views
Setting up a Rails 2.3.x app on EC2 for easy scalability
I'm running a simple rails stack on a single dedicated machine. We're reaching our full capacity and have absolutely no setup for scaling, just one app on one machine. I did some research and came up ...
- 670
3
votes
2
answers
600
views
Distributed, Parallel, Fault-tolerant File System with high throughput
I am looking for DFS (distributed file system) that is fault tolerant and easy to maintain. I will have tons (100M+) of small files (from 1K to 500K). Files will be located in some directories what ...
- 647