3

I have a single ZFS/Linux instance with a file that can grow to those crazy sizes supported by ZFS.

A large number of concurrent users need to read/write from/to this file over the public inter net. How do I load-balance this massive file-access load across a pool of ZFS/Linux instances with identical file-system data and meta-data?

Also - While ZFS can self-heal from minor bit-rot, how can failover to a replicated set of ZFS/Linux instances in case of a total hardware failure be implemented?

Nobody seems to be talking about replicas, failovers, and load-balancing in context of ZFS. Or maybe, I haven't trawled the Internet long enough.

9
  • 1
    If you want to hear my opinion, the application is just trash and poorly designed. What kind of an application depends on a single gigantic file?!
    – paladin
    Feb 21 at 13:37
  • @paladin Fine, ignore for a moment the case of a single gigantic file. How about a large no. of small files spread across various directories in the ZFS instance? File-locking for concurrent access is also taken care of. How do I now address failover and load-balancing for this ZFS instance?
    – Harry
    Feb 21 at 14:00
  • 2
    This entire question is missing lots of details and concrete concepts. For example, does "instances" refer to disks, zpools or whole server nodes? How does "read/write" happen, over NFS, SMB or HTTP (like WebDAV)?
    – iBug
    Feb 22 at 10:27
  • 1
    ZFS is designed as a single-host storage solution, so (cross-node) replication, failover, clustering and other high-availability features are never a concern of ZFS. Chances are you'll have to handle all of these at the application level, which is why I asked about file protocol in the first place. If you really need that kind of availability that a single enterprise-grade server can't provide, you should look into things like Ceph, but be warned about their performance.
    – iBug
    Feb 22 at 10:29
  • 1
    I think there's a big XY problem here. "A large number of concurrent users need to read/write from/to this file over the public inter net." tells me "web services backed by a database" not "gigantic shared file". Then the issue of what type of database (SQL or NoSQL) and what kind of replication/load balancing comes into play, but it's very unlikely that this would be a FS problem.
    – jcaron
    Feb 22 at 11:32

2 Answers 2

6

Neither ZFS nor BTRFS are intended to be replicated between instances in real time. You need to look into the specific category of Distribute File Systems. The Wikipedia page Comparison of distributed file systems gives a good starting list.

5
  • When you create a BTRFS RAID1 array with 2 volumes, you can later add 2 more volumes and increase RAID1 to RAID1c4, which will create 4 complete copies of each file, 1 copy per volume. This works all online and doesn't need much administrative work. What do you mean with real time?
    – paladin
    Feb 22 at 10:21
  • @paladin the main words are between instances.
    – AlexD
    Feb 22 at 10:22
  • @AlexD It's not even clear what "instances" refers to. Disks? Pools? Server nodes? Without that we can only speculate and can't work out an answer.
    – iBug
    Feb 22 at 10:22
  • @iBug the question mentions ZFS/Linux instances, failover to a replicated set of ZFS/Linux instances in case of a total hardware failure.
    – AlexD
    Feb 22 at 10:26
  • @iBug I've clarified it now that I mean the entity in ZFS that abstracts away all disks, zpools and the ZFS-specific what-have-yous that a C application program will never want to care to know about when using fread or fwrite.
    – Harry
    Feb 27 at 5:09
1

I don't know much about ZFS "RAID" modes, but BTRFS "RAID" modes are unique and special and grant great flexibility, inbuilt-load-balancing and redundancy.

For example BTRFS RAID1 doesn't work on the block level, but on the filesystem level. When you create a BTRFS RAID1 with 4 disks, the RAID mode says that every file is at least stored on 2 different disks (on filesystem level). So for example you have disks A, B, C and D, now you save your file TESTFILE onto your BTRFS RAID1, TESTFILE is then saved on 2 of the 4 drives as a complete copy. This gives you an performance advantage, as only 2 drives really need to do work in that moment, same goes for reading files.

With BTRFS you can just add/remove disks to/from the "RAID"-array without any problems. You can replicate entire BTRFS filesystems or just the subvolume parts of filesystems using the send/receive commands (useful for backups).

I really don't know much about ZFS but I could imagine that it has also such feature like BTRFS. If it has not such feature, I would replace it with BTRFS.

9
  • It is faster to read a file in parallel from 4 drives than from 2, so the words "same goes for reading files" are incorrect.
    – AlexD
    Feb 22 at 10:41
  • @AlexD If only that happens with BTRFS, which load-balances read by PID. Interesting approach though often not the most performant.
    – iBug
    Feb 22 at 10:43
  • @iBug, do I understand correctly that BTRFS doesn't benefit from striping and reads only from one stripe?
    – AlexD
    Feb 22 at 10:54
  • @AlexD "Striping" is like RAID 0. My previous comment applies to BTRFS implementation of RAID 1 or mirroring. Also if you have multiple processes working together you do get the benefit.
    – iBug
    Feb 22 at 11:29
  • Imagine you have 4 users, each user is accesing different files from the same BTRFS filesystem. The filesystem is a RAID1 (BTRFS style) of 4 volumes. The 4 different files are stored across the RAID1, each file has a double on another volume. Volume A has File1 and File2, volume B has File2 and File3, volume C has File3 and File4 and volume D has File4 and File1. User A access File1, user B access File2, user C access File3, user D access File4 --> volume A serves File1 to user A, volume B serves File2 to user B, volume C serves File3 to user C, volume C serves File4 to user D, independently.
    – paladin
    Feb 22 at 11:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .