Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [distributed-filesystems]

The tag has no usage guidance.

0
votes
0answers
54 views

Gluster: how to setup distributed replicated volume with different size

Say I have 4 servers, each with ZFS volumes. Two servers with 36GB each. And two servers with 60GB each. I want to use glusterfs to setup two replicated volumes between the similar sized ZFS volumes. ...
0
votes
1answer
111 views

glusterfs client mount point fails with 'Client quorum is not met'

Assume a basic replica 3 arbiter 1 configuration, glusterfs-server 4.0.2 and glusterfs-client 4.0.2. glusterfs-client is installed on Ubuntu 18.04. In order to verify that write / read ops are ...
2
votes
1answer
246 views

Does Perkeep (camlistore) have built in protection from bitrot?

Does Perkeep (PKA Camlistore) offer protection from silent corruption (eg. bitrot) of the data in its current design like what is offered in ZFS. If it does, how well does it fare when compared to ZFS?...
1
vote
1answer
822 views

DFS Namespace unavailable at login

Background We are having some issues with DFS namespaces in our environment after a change to how we configured our environment and I was looking to get some insight on how to troubleshoot, before we ...
2
votes
1answer
107 views

Which distributed filesystem will actually fit my needs?

Hear me out I have seen the question asked (in different forms) here, here, and perhaps the best one I found was here, but I do not think this is a duplicate because quite some time has past since ...
1
vote
0answers
108 views

Is it practical to keep home directories on OpenAFS?

I am a sysadmin by trade, and I do what I do at work at home as well for fun. I have a Gentoo Linux laptop, Raspberry Pis running Raspian, a Gentoo server, ARM devices running Debian and have various ...
0
votes
2answers
360 views

Networked In-Memory FileSystem w/ zram + XFS + GlusterFS - Best Practices

A few months ago I used XFS formatted zram devices strung together with GlusterFS to create a distributed / networked / replicated in-memory filesystem on a few bare metal servers (running RHEL 7.2). ...
0
votes
1answer
539 views

Asynchronous file replication over long distances, with buffering

I'm looking for a filesystem that can replicate over long distances, and tolerate being offline for extended periods of time by using a local buffer, which should be a disk buffer, to queue up changes ...
1
vote
1answer
227 views

How to make GlusterFS mount point highly available?

It's true that one can mount a GlusterFS volume on any brick and the FUSE client will automatically discover the other bricks and connect to them as well. If the initial brick fails the client will ...
0
votes
1answer
44 views

GlusterFS server logic

I quite don't understand about GlusterFS probes. See, i add all servers to /etc/hosts file and i do this for each server. Lets say i have 100 servers and later i added 100 more. I cannot imagine ...
2
votes
1answer
7k views

GlusterFS Transport endpoint not connected from time to time

I'm using GlusterFS 3.7.9, currently on a single server with 4 bricks. Each brick has 4TB and the volume is set up as distribute only. The volume is mounted on a secondary server and I use it for ...
4
votes
2answers
412 views

GlusterFS alternative for file upload website

I have a few file upload websites, with files ranging from hundreds of kilobytes to a few gigabytes. Currently I have all files in a distribute-replicate Gluster volume on a few servers. My ...
1
vote
1answer
181 views

Glusterd dead but cluster still running

We are currently running a three node cluster, on Gluster 3.6.4. On one of our nodes we noticed that the glusterd daemon is dead. But the glusterfsd daemons are still running, and we believe clients ...
1
vote
0answers
17 views

Is it possible to configure hdfs in a federation mode and in an HA mode in the same time?

I don't understand if it possible to configure HDFS in both modes in the same time. Does it make sense? Can somebody show a simple configuration of HDFS in both modes? (nameNode1, nameNode2, ...
1
vote
1answer
290 views

How to safely store data and DBs on Kubernetes (or similar) on DigitalOcean

It boils down to setting up a private cloud on something like Digital Oceans versus another hosting solution like GCE or AWS. The latter have redundant data stores that can be safely used as volumes, ...
0
votes
1answer
603 views

New files on Linux Samba share: Now you don't see me, now you do

When a file is uploaded to our server, we have to do some work on it to allow or reject it. Some files need to be opened in a Windows box (OpenXml Sdk performance reasons). Our web server runs in a ...
2
votes
1answer
83 views

distributed storage solutions [closed]

I've got some desktop PCs available in my company since the last user hardware renewal and I would like to re-use them for a demo plaform. They are powerfull enought to emulate a few servers. I ...
1
vote
0answers
447 views

Glusterd won't start after node crash

We had a 2 node cluster running GlusterFS. It's a 4 volume cluster, replica 2, one brick per server. Dead server: nnfs02 Live server with all data: nnfs03 New server: nnfs01 After nnfs02 died, we ...
6
votes
2answers
472 views

What is reasonable storage failover time that most OS (VM) can tolerate?

I have a GlusterFS 2 node 2 replica setup. I am planning to use it as OpenStack instance store, in which the VM disk image is stored. From my tests, if the GlusterFS node which the hypervisor ...
3
votes
1answer
943 views

Distributed mirrored filesystem under FreeBSD

Can someone share their experience in building a distributed mirrored filesystem between multiple FreeBSD machines? I. e. we have two (three, four...) servers and special partition "part1" mounted on ...
0
votes
1answer
249 views

Expanding/Growing OCFS2 forever

I have a kvm environment where a shared disk mounted by 2 VMs through rbd ceph. What I would like to know that in the future how is it possible to expand OCFS2 filesystem what I planning to use on top ...
0
votes
1answer
282 views

Storing Large Amount of files in a file system?

Guys I am creating an application which will be saving a .txt file (approx 10 KB/file I guess) in a file system and will be saving its path in MS SQL database. Now as per my estimation, approximately ...
1
vote
3answers
3k views

LizardFS/MooseFS — Multiple Storage Volumes

I'm using the new LizardFS and it's awesome! I'm new to LizardFS/MooseFS overall, but it seems pretty straight-forward. I was wondering two things: First, is there a way to setup separate storage ...
5
votes
2answers
1k views

Is there a way to connect a remote folder with local cache on OSX or Windows workstation?

A company with 50 graphic designers are working on the same file server. They are using mainly Indesign. A typical project is a 60 pages indesign document with 1.5GB of linked files (something like ...
3
votes
1answer
3k views

Why is GlusterFS so slow here?

We've set up a mirroring pair of GlusterFS servers. No special tuning, whatever came "out of the box" with GlusterFS-3.5.1 in the official RHEL6 RPM, that's what we have. The cluster works, but the ...
6
votes
0answers
161 views

Fast distributed filesystem for a large amounts of data with metadata in database [closed]

My project uses several processing machines and one storage machine. Currently storage organized with a MSSQL filetable shared folder. Every file in storage have some metadata in database. Processing ...
1
vote
2answers
203 views

Setting up a multi-host SAN

I'm looking for some advice on setting-up multi-host I/O access on our SAN: I have a blade enclosure (PowerEdge1000e) containing an Equallogic PS-M4110 storage blade with a single RAID6 volume ...
1
vote
2answers
425 views

Simple, decentralized, automatically synced, local cached file share for linux server

I wonder if someone could recommend me some solution to sharing files between a small amount of linux VPS (virtual private servers). Basically I have a bunch of linux VPS to manage, and I want to ...
3
votes
2answers
2k views

what distributed file system for a two-node failover setup?

I'm trying to set up a redundant setup consisting of two servers that have everything redundant: the database (MySQL master-master in active/passive mode) the file system (distributed/replicated) ...
3
votes
1answer
2k views

In a distributed filesystem like MooseFS or XtreemFS, should individual nodes expose “raw-er” storage, or LVM'd storage?

When preparing an infrastructure to utilize a distributed storage system like MooseFS or XtreemFS, how should individual nodes present storage to the rest of the environment? Is it better for ...
1
vote
2answers
1k views

Scale-out distributed storage with snapshots

I'm aware there are many similar questions around, several with great answers. I still haven't quite found what I'm looking for though: a distributed, scale-out FS that supports snapshots. Gluster ...
0
votes
2answers
74 views

Distribute all services between 3 VPS [closed]

i just rent 3 vps servers with this: 4GB RAM 150GB hard disk 2 CPU 6 ip address so we have vps1, vps2 and vps3 and i'm thinking distribute the services like this: VPS1 - frontend + DNS Server + ...
0
votes
3answers
408 views

File system benchmark that generates I/O from distribution of file sizes [closed]

I am looking for a file system benchmarking tool to test CephFS. This benchmarking tool has to: Generate a set of files (call it fileset) following a particular distribution, e.g., gamma, empirical, ...
5
votes
8answers
4k views

Distributed file system with local disk cache

My server infrastructure is growing fast and I decided to create a distributed storage cluster. I've been looking for a proper filesystem for this task which meet my requirement, but none of them ...
1
vote
2answers
236 views

Distributed storage and computing [closed]

Dear Serverfault community, After researching a number of distributed file systems for deployment in a production environment with the main purpose of performing both batch and real-time distributed ...
3
votes
3answers
385 views

Choice of distributed filesystem for intensive reads and writes

I have a series of servers (HP ProLiant, 34 servers) each of which with 500 G of hard drive space. These servers are part of a computational cluster which runs processes that roughly fall into two "...
2
votes
1answer
475 views

Fault-tolerant network access to redundant storage in a heterogeneous environment (including Windows)?

We are facing a design "challenge" where we are required to set up a storage solution with the following properties: What we need HA a scalable storage backend offline/disconnected operation on the ...
1
vote
3answers
540 views

Distributed filesystem for storing temporary files

We have a cluster of web servers that currently stores temporary files (that have been uploaded by users) on a central shared filesystem. The limitation being that anytime the server containing the ...
1
vote
0answers
552 views

How do I speed up “ls -l” on a network file system (NFS)?

I have a NFS volume mounted in the following way: theHostname.com:/path/on/theHostname on /local/path type nfs (rw,nosuid,addr=10.0.0.54) The /local/path directory has 30_000 files. When I do a ls ...
2
votes
0answers
2k views

NFS showing files in directory, but can't open or stat

I'm using a network of Linux (Debian Squeeze on kernel 2.6.32) machines, sharing files using NFS (v3). The scenario is that a process running on client A will create a file through NFS on file server ...
2
votes
1answer
57 views

distributing access across multiple disks

I have a rather large application that requires simultaneous access to rather a lot (300+) of large files, and accesses them by reading them all at the same time. If all the files are put on 1 disk (...
0
votes
1answer
125 views

Migration from ext3 to Mogilefs

I have gigs of images on ext3 file system and I want to move them to Mogilefs on a live running site. In your opinion what would be the best approach?
0
votes
1answer
118 views

In search of a distributed file-level filesystem [closed]

I'm looking for a distributed, fault-tolerant file-level (not block-level, preferably) filesystem for use in a small Linux-based HA cluster of sorts. The files that are stored will be fairly small (I'...
3
votes
0answers
589 views

Are snapshots and clones filesystem-wide in ZFS-backed Lustre clusters

My goal is to find a distributed filesystem on Linux that supports ZFS-like lightweight snapshots and snapshot clones. This StackOverflow question expresses what I'm looking for pretty well. I'm ...
4
votes
2answers
255 views

Distributed storage [closed]

At my University Department we are about to upgrade the computers of our student lab (about 25-30 machines). The machines will be running Linux. One thing about the new machines is that they have ...
16
votes
4answers
23k views

Are ZFS clustered filesystems possible?

Is it possible to create a ZFS cluster? Or would you need to go the seemingly ugly (to me at least) route of ZFS with UFS formatted pools governed by GlusterFS? The idea is to see if it is possible ...
0
votes
1answer
142 views

SQl Server sharding

Can SQL Server be sharded for distributed data processing? Or are there any limitations that prevent relational databases from being sharded ?
2
votes
3answers
1k views

failsafe RAM drive solution [closed]

I'm looking for a production solution to create a RAM drive that will be safely synchronized with HDD. I have a piece of custom software with heavy I/O load (this is some kind of proprietary document-...
1
vote
1answer
537 views

DFS - Stop sync of large folder that has since been removed

We have a site to site DFSR on Windows Server 2008 R2 that has been running perfectly between site A to site B until someone dumped a 20GB folder. This has overwhelmed the upload and make the internet ...
3
votes
1answer
840 views

Distributed filesystem for automated offline data mirroring [closed]

I'd like to achieve the following setup: Every time I connect my laptop to a local network, my partition gets automatically mirrored to a partition on my local server. I only want to mirror what has ...