3
votes
1answer
51 views

btrfs Integrity and Stress Testing

A DevOps colleague is recommending that we begin transitioning our production environment to using btrfs. We have primarily ext4 filesystems, though some low-usage servers using ZFS (on Linux). As ...
-2
votes
1answer
96 views

Shared storage solution for Debian Wheezy [closed]

I need to setup a new infrastructure which provides high available shared storage between debian servers. Our current solution is a "clustered DRBD NFS solution". We have setup 2 virtual file ...
0
votes
2answers
43 views

linux minimal block size [closed]

Question : I would like to know if one could create an empty partition with a very small block size, say 100 bytes, and get their experience on the long run. Are there any gotchas ? Context : I ...
0
votes
1answer
261 views

btrfs: Looking for experience on btrfs error modes on bad sectors / read errors / medium erros

While running a Hadoop cluster that uses HDFS (so data is already replicated 3x) we experience often issues with ext4 because some bad sectors (the filesystem is unable to read or write to them) ...
7
votes
1answer
242 views

Which block sizes for millions of small files

I have 2x 4TB Disks in hardware RAID1 (it might be a LSI MegaRaid) on Debian Wheezy. The physical block size is 4kB. I'm going to store 150-200 million of small files (between 3 and 10kB). I'm not ...
1
vote
1answer
692 views

How often should I defrag ZFS pools?

I ZFS scrub once a month which takes ~24 hours for the 72TB. For what I can find about how often I should scrub is: very busy pools, once per week not so busy pools, once per month or run a scrub, ...
-2
votes
1answer
49 views

What tool/application (free or not) should I use to generate and write lots (TBs) of data, fast [closed]

I am testing a storage appliance and need to write TeraBytes of data to it. Using fio or dd takes days to write that much amount of data. Are there any (free or not-free) tools/applications, that I ...
4
votes
3answers
204 views

Store multiple versions of large binary file with minimal data duplication (preferably Linux)

I need to store multiple versions of a ~ 150 GB binary file (qcow2) on Linux servers with local storage, and was hoping there is some solution that involves just keeping diffs that can be merged as ...
-1
votes
1answer
70 views

Storage virtual machine. LVM or loop devices? [closed]

Actual situation: I'm the administrator of a virtual machine. The virtual machine is a Debian. This machine will be mainly a storage machine with different services (svn, ftp... ), and a webserver. ...
3
votes
1answer
1k views

Choosing file system block size

Considering unix/linux/bsdunix specific file systems: How can I choose/know which block size to use while creating file system? Is there any specific block size value for a particular file system ...
12
votes
2answers
827 views

Safety of write cache on SATA drives with barriers

I've been reading lately about write caching, NCQ, firmware bugs, barriers, etc regarding SATA drives, and I'm not sure what's the best setting that would make my data safe in case of a power failure. ...
5
votes
3answers
474 views

What is the ZFS ACL limit?

Does anyone know how many user ACL's ZFS can handle? With other words: for how many users can I set ACL's like this one for the same directory? setfacl -m user:test1:rwxpDdaARWcCos:fd----:allow ...
2
votes
1answer
181 views

Why is ZFS replication single threaded?

On my 8 core NAS I see only one core maxed out when ZFS replication over SSH is running. I would expect that transferring a snapshot would scale perfectly with the number or cores, as the snapshot ...
0
votes
1answer
68 views

4million+ images on a single LAMP stack server

Currently I am trying to come up with a naming convention for files and folders, essentially a method of spreading out 4 million+ images on a server into various directories or whatever. What I am ...
3
votes
1answer
95 views

Storing valuable high-resolution images before root

I have a busy celebrity stock photo website and I am trying to hide the photos behind the website root, so they can't be accessed via URL, but my ASP script can read the photo and sends it to he ...
-2
votes
4answers
420 views

How can I create one big volume from multiple drives, but loose only the files stored on one disk if it fails?

I am running a storage system. Fault tolerancy is handled at application level (files are stored on multiple servers, even datacenters). Therefore I don't need nor even want redundancy within one ...
2
votes
3answers
402 views

Possible to per file encrypt cloud storage? [closed]

With the many cloud storage providers that there now exist, would it be possible to add a layer of encryption before the files are saved? It would be possible to make an encrypted partition in a ...
1
vote
2answers
416 views

ZFS: Best-practices doc state: “On systems with 1,000s of ZFS file systems…”?

The docs state: "On systems with 1,000s of ZFS file systems, provision 1 GB of extra memory for every 10,000 mounted file systems including snapshots." What does that mean - how do these people use ...
2
votes
1answer
115 views

Better folder structure for 1-6mil files?

I'm doing a bit of development work for a start up, and have been asked to work on the directory structure for all the content being served and the scalability of the host servers(load balancing, ...
-4
votes
2answers
1k views

Does the number of files slow down the server?

Having numerous files will obviously slow down the OS; but how much this problem is serious? Consider that the files have been well distributed over multi-level folders. Does still the number of files ...
2
votes
1answer
47 views

Is it possible to import file system owned by a single user and export as multiple virtual volumes, owned by multiple users?

Is it possible to import a file system owned by a single user, (e.g. mounted via NFS) and export it again as multiple virtual volumes, owned by multiple users, where each volume is a folder on the ...
23
votes
12answers
3k views

Storing and backing up 10 million files on Linux

I run a website where about 10 million files (book covers) are stored in 3 levels of subdirectories, ranging [0-f]: 0/0/0/ 0/0/1/ ... f/f/f/ This leads to around 2400 files per directory, which is ...
1
vote
3answers
13k views

For sharing storage between virtual machines using ESXi should I use shared virtual drives or a shared physical hard drive?

I want to store media assets and such that can be accessed by multiple virtual machines as a windows drive path. The question is can I (and should I): A) Have a physical hard drive partitioned to ...
0
votes
2answers
569 views

Best filesystem for VM storage? [closed]

What is the best file system for storing VM images? I am looking for features that can save some disk space like compression, deduplication, etc. ZFS is known...but any other alternatives in Linux ...
4
votes
5answers
1k views

ZFS on Linux based systems

Is there anyone who has more experience with ZFS under Linux based systems ? I saw ZFS fuse library, but I don't know it;s worth a test or not. Stable, usable ? Maybe as a storage ? Or only ...
0
votes
1answer
352 views

Getting QFS for Solaris 10

I am trying to get the QFS FS which comes along with SUN Storage solutions, http://www.oracle.com/us/products/servers-storage/storage/storage-software/031712.htm But the download link on that page ...
0
votes
2answers
126 views

Cost of storing many files on a fileserver

I need to store a lot of data contained in SQLite databases on a file server. I have the opportunity to split the data up into many files. This means that there is less risk of a large part of the ...
1
vote
1answer
232 views

Generating error correction data for existing files (backups)

My server has a big dataset stored on disk. I am worried by the possibility of the data being silently corrupted over the years without anybody noticing. My idea was to generate "recovery data" for ...
2
votes
1answer
649 views

Recommended technology for layered disk cache in linux

I originally asked this on superuser but it occurs to me it's more more of a server topic. I've just bought a 6-core Phenom with 16G of RAM. I use it primarily for compiling and video encoding (and ...
10
votes
5answers
1k views

Tips on efficiently storing 25TB+ worth million files in filesystem

Say you are confronted with 25 TB worth uncompressed log files and you have at your disposal an array of 20 commodity boxes with collective free storage capacity of 25 TB. How would you store these ...
0
votes
2answers
336 views

most efficient way to expose storage to windows on network?

using Nexenta or Opensolaris...on server grade h/w: SAS disks + 1GB layer3 switch: CIFS would only do 10 MB/s. Adding more servers would bump this to 10 MB/sec per server? dont know about NFS ...
3
votes
2answers
159 views

Making many network shares appear as one

Givens: disk is cheap, and there's plenty lying around on various computers around the corporate intranet redundant contiguous large storage volumes are expensive Problem: It would be fantastic ...
0
votes
2answers
593 views

What is the best filesystem for storing thousands of files in one dictionary-like id-blob structure?

What filesystem best suits my needs? Thousands or even millions of files in one directory. Good (ext4 & ntfs level or close) reliability (incl. fault tolerance) and access speed. No directories ...
2
votes
4answers
2k views

multiple file systems for mysql

Does mysql support multiple file systems for a single database with most of the tables being on MyISAM? Context: we have a 1.5TB mysql database, which is increasing at the rate of 200GB per month. ...
0
votes
2answers
433 views

Which CDN do you recommend for good organization of files?

I have many projects that need to be organized in sub-buckets and sub-sub-buckets. Which CDN do you recommend? I asked this question earlier, and some people told me to "fake" S3: ...
2
votes
5answers
565 views

Choosing a NAS system and things to watchout for

We are looking at implementing network attached storage to host some our DFS shares. Who are the good / bad vendors? Has anybody already tried this and did it work for them are there better ...
16
votes
10answers
9k views

Distributed Storage Filesystem - Which one/Is there a ready to use product?

With Hadoop and CouchDB all over in Blogs and related news what's a distributed-fault-tolerant storage (engine) that actually works. CouchDB doesn't actually have any distribution features built-in, ...
1
vote
6answers
272 views

What's a reasonably secure web-based storage system, that ideally can be mounted as a virtual filesystem on Windows?

Our small community college assigns each of our approximately 1000 students a personal My Documents folder that is an alias for a Windows server share (also mapped as their P: drive). Students access ...
10
votes
8answers
10k views

Is it possible to hide lost+found?

Given a partition intended solely for storing music, video and so-on, is it possible to hide the lost+found directory?
32
votes
11answers
21k views

ZFS vs XFS

We're considering building a ~16TB storage server. At the moment, we're considering both ZFS and XFS as filesystem. What are the advantages, disadvantages? What do we have to look for? Is there a ...