34

I have recently looked into advanced filesystems (Btrfs, ZFS) for data redundancy and availability and got interested in the additional functionality they provide, especially their "self-healing" capabilities against data corruption.

However, I think I need to take a step back and try to understand if this benefit outweighs their disadvantages (Btrfs bugs and unresolved issues & ZFS availability and performance impact) for general home/SMB-usage, compared to a conventional mdadm-Raid1 + Ext4 solution. A mirrored backup is available either way.

Let's assume I have a couple of file servers which are used for archival purposes and have limited resources, but ECC memory and a stable power source.

  1. How likely am I to even encounter actual data corruption making files unreadable? How?
  2. Can Ext4 or the system file manager already detect data errors on copy/move operations, making me at least aware of a problem?
  3. What happens if one of the madam-Raid1 drives holds different data due to one drive having bad sectors? Will I still be able to retrieve the correct file or will the array be unable to decide which file is the correct one and lose it entirely?
9
  • 10
    Btrfs is a nightmare and will go read-only if it runs out of metadata space, go zfs if you follow that route May 17, 2020 at 1:51
  • I'm aware of that. Reading about its issues on the Debian Wiki made me rethink my plans of using it and open this reflective question here. Unfortunately ZFS is not in the Linux Kernel and won't be at least for a while according to Linus Torvalds, making it cumbersome to set it up after a change of systems and riskier in case of an emergency. May 17, 2020 at 12:49
  • 1
    Read up on XFS? May 17, 2020 at 12:56
  • 2
    @Prototype700 ZFS is in Ubuntu. If you are setting up a NAS system, either use something Ubuntu based or something FreeBSD based like the great TrueNAS (currently still called FreeNAS) system, and you won't have any problems.
    – Josef
    May 17, 2020 at 20:00
  • 1
    @Timothy Frew, I did, XFS doesn't appear to have any more data protection features than Ext4, just faster preformance, if I haven't missed anything? May 18, 2020 at 10:25

7 Answers 7

33

Yes, a functional checksummed filesystem is a very good thing. However, the real motivation is not to be found into the mythical "bitrot" which, while does happen, is very rare. Rather, the main advantage is that such a filesystem provide and end-to-end data checksum, actively protecting you by erroneous disk behavior as misdirected writes and data corruption related to the disk's own private DRAM cache failing and/or misbehaving due to power supply problem.

I experienced that issue first hand, when a Linux RAID 1 array went bad due to a power supply issue. The cache of one disk started corrupting data and the ECC embedded in the disk sectors themselves did not catch anythig, simply because the written data were already corrupted and the ECC was calculated on the corrupted data themselves.

Thanks to its checksummed journal, which detected something strange and suspended the filesystem, XFS limited the damage; however, some files/directories were irremediably corrupted. As this was a backup machine facing no immediate downtime pressure, I rebuilt it with ZFS. When the problem re-occured, during the first scrub ZFS corrected the affected block by reading the good copies from the other disks. Result: no data loss and no downtime. These are two very good reasons to use a checksumming filesystem.

It's worth note that data checksum is so valuable that a device mapper target to provide it (by emulating the T-10 DIF/DIX specs), called dm-integrity, was developed precisely to extend this protection to classical block devices (especially redundant ones as RAID1/5/6). By the virtue of the Stratis project, it is going to be integrated into a comprehensive management CLI/API.

However, you have a point that any potential advantage brought by such filesystem should be compared to the disvantage they inherit. ZFS main problem is that it is not mainlined into the standard kernel, but otherwise is it very fast and stable. On the other hand BTRFS, while mainlined, has many important issues and performance problem (the common suggestion for databases or VMs is to disable CoW which, in turn, disabled checksumming - which is, frankly, not an acceptable answer). Rather then using BTRFS, I would use XFS and hope for the best, or using dm-integrity protected devices.

4
  • Thank you for the detailed explanation and the suggestions. Could you elaborate a bit more on how the XFS checksummed journal is able to detect the corruption? (Is this done automatically online or on a manual check? Can the corrupt directories/files be identified?) And, if you can, where the limits of this feature are compard to having an end-to-end data checksum? I think, while each exact use-case has to be considered, maybe this could be a viable middle-ground option if corruption can be detected and e.g. a versioned backup is readily available (and if auto-correction is not of importance). May 18, 2020 at 22:55
  • 2
    XFS journal and metadata checksum is performed on-line: if a serious error is discovered, the filesystem is suspended. Anyway, XFS behavior can be (partially) fine-tuned by the sysfs entries found in /sys/fs/xfs/<blockdev>/error/. For more information, I suggest you reading XFS docs and/or asking on the mailing list.
    – shodanshok
    May 19, 2020 at 12:06
  • The word "bitrot' often encompasses not just magnetic domains spontaneously changing, which may be rare, but also all the other sources of silent corruption including bad DRAM, controller failures, etc.
    – poolie
    May 21, 2020 at 19:07
  • As the answer hasn't made it very clear: Please note that XFS checksums 1) are a feature only in newer XFS versions (no problem), and 2) do not protect data (they protect only metadata) (big problem). Therefore, I wouldn't use it, or would use it only on top of dm-integrity.
    – Binarus
    Jan 3, 2021 at 9:36
11
  1. I had a Seagate HDD that started failing checksums each time I was running zfs scrub. It failed after a few weeks. ZFS and Btrfs have checksums for data and metadata. ext4 has only metadata chcksums.

  2. Only CRC errors and metadata checksum errors. Data corruption can happen.

  3. If it has bad sectors it is not a problem. The entire disk will be "failed", but you have the other disk that is "fine". The problem is when the data has correct CRC, but the data is corrupted. This can happen randomly because of large disks.

11
  • 1
    Ext4 is doing metadata checksums to have a safety net and not interpret some random data as file system objects. Better to return an error than some garbage. May 18, 2020 at 12:31
  • 1
    On a storage system you need a lot of redundancy and verifications. This is done via RAID (1, 5, 6, 10, Z...), line code (e.g Manchester), checksums (crc, sha256), snapshot, backups. Some techniques are used multiple times for same data (e.g. CRC on disk and SHA on ZFS), duplicate of some metadata in file system and duplicate of the entire disk in RAID and some redundancy caused by line coding. May 18, 2020 at 12:42
  • 3
    @Prototype700 The point of metadata checksums is that unlike data checksums, they can be retrofitted to non-CoW filesystems and they still protect against the worst kinds of corruption. While the contents of any given file isn't protected, filesystem structures are, so the rest of the filesystem will keep working and the damage will be confined to the directly affected files. Without metadata checksums, damage to metadata may lead to silently growing corruption of the core filesystem structures (which blocks belong to files and which ones are free) until the whole FS is damaged beyond repair.
    – TooTea
    May 18, 2020 at 15:35
  • 1
    @Prototype700 Also, many hardware failures will trip metadata checksums as well, alerting you to the fact that some corruption is happening. You won't know which files are corrupted so you'll have to do a full restore, but you will know that something went wrong before all your backups contain corrupted data as well.
    – TooTea
    May 18, 2020 at 15:37
  • 1
    Usually file systems will re-mount read-only when a corruption is detected and will record the problem in kernel log. Kernel log can be send to a remote machine for centralized logging. May 18, 2020 at 23:09
9

I have been using ZFS in production, for both servers and a home office NAS, under both Linux and FreeBSD, for over 6 years. I have found it to be stable, fast, reliable, and I have personally seen it detect and (when able to) correct errors which a simple md device or ext4 filesystem would not have been able to.

However, I think I need to take a step back and try to understand if this benefit outweighs their disadvantages (Btrfs bugs and unresolved issues & ZFS availability and performance impact)

Regarding licensing, ZFS is open source it's just released under the CDDL license which is not legally compatible with the GPLv2 license that the linux kernel is released under. Details here. This does not mean it's in a state of "lincensing-limbo for a while" nor does it mean there's any technical incompatibility. It simply means the mainline linux kernel source doesn't have the modules and they have to be retrieved from somewhere like https://zfsonlinux.org . Note that some distros, like debian, include ZFS in their distribution Installing ZFS on Debian / Ubuntu can normally be done with a single apt command.

As for performance, given sufficient RAM ZFS performance for me is anywhere from close to ext4 to surpassing ext4, depending on memory, available pool space, and compressibility of data. ZFS's biggest disadvantage in my opinion is memory usage: If you have less than 16 GiB of RAM for a production server, you may want to avoid ZFS. That is an overly-simplified rule of thumb; there is much information online about memory requirements for ZFS. I personally run a 10TB pool and an 800GB pool along with some backup pools on a home office linux system with 32GB RAM and performance is great. This server also runs LXC and has multiple services running.

ZFS features go well beyond the data checksumming and self-healing capabilities; its powerful snapshots are much better than LVM snapshots and its inline lz4 compression can actually improve performance by reducing disk writes. I personally achieve a 1.55x savings on the 10TB pool (storing 9.76GiB of data in only 6.3GiB of space on disk)

In my experience, ZFS performance degrades when the pool reaches 75% or 80% usage. So long as you stay below that point, performance should be more than sufficient for general home/SMB-usage.

In the cases I have seen ZFS detect and correct bad data, the root cause was unclear but was likely a bad disk block. I also have ECC memory and use a UPS, so I don't believe the data was corrupted in RAM. In fact, you need ECC RAM to get the benefit from ZFS checksums. However I have seen a handful (~10-15) cases of blocks which failed checksums over the past 6 years. One major advantage of ZFS over an md RAID is that ZFS knows which files are affected by a checksum error. So in cases where a backup pool without redundancy had a checksum error, ZFS told me the exact files which were affected, allowing me to replace those files.

Despite the license ZFS uses not being compatible with the linux kernel, installing the modules is very easy (at least on Debian) and, once familiar with the toolset, management is straightforward. Despite many people citing fear of total data loss with ZFS on the internet, I have never lost any data since making the move to ZFS, and the combination of ZFS snapshots and data checksums/redundancy has personally saved me from experiencing data loss multiple times. It's a clear win and I'll personally never go back to an md array.

7
  • Why do you say you need ECC to get any benefit from ZFS checksums? Just curious.
    – user35915
    May 18, 2020 at 7:46
  • Hi Josh, to comment on your points: "I am not sure what the ZFS "availability" impact you reference is" - The fact that it is and will be in a state of lincensing-limbo for a while and the head of Linux waived any chances of including it in the kernel. Since distros can't add it to their main (free) repositories, what exactly you get might change depending on the system, repository, etc. Might become relevant in an emergency or when a system has to be changed without wanting to change the storage. May 18, 2020 at 10:46
  • On the question of "performance" - Here, the issue is that ZFS seems great for when nothing else is run on the server, but at home, I don't have multiple servers running, so I might need resources for additional uses at the same time. Also, the RAM requirements seem quite demanding, since small home servers will likely be on that edge already, depending on the exact size and hardware configuration. The same might be the case in small businesses when the server is not just a file storage. I know that FreeNAS can also run additional services, but still ZFS seems a lot more demanding than Ext4. May 18, 2020 at 10:54
  • @user35915 short story: Because zfs relies on the RAM to calculate checksums, so bad RAM can make bad checksums. Details in this question
    – Josh
    May 18, 2020 at 12:46
  • @Prototype700 Thanks for clarifying! I edited and addressed these points. The license issue has never been a problem for me: a fresh debian install can have ZFS running in about 5 minutes after install, and the oldest ZFS pool I use has been transferred between multiple systems over the past 6+ years with no issues whatsoever.
    – Josh
    May 18, 2020 at 12:48
4

How likely am I to even encounter actual data corruption making files unreadable? How?

Given enough time, it's almost certain to happen. Coincidentally, it happened to me last week. My home file server developed some bad RAM that was causing periodic lockups. Eventually I decided to simply retire the machine (which was getting rather old) and moved the drives to an enclosure on a different machine. The post-import scrub found and repaired 15 blocks with checksum errors, out of an 8TB pool, which were presumably caused by the bad RAM and/or the lockups. The disks themselves had a clean bill of health from SMART, and tested fine on a subsequent scrub.

Can Ext4 or the system file manager already detect data errors on copy/move operations, making me at least aware of a problem?

No, not really. There might be application-level checksums in some file formats, but otherwise, nothing is keeping an eye out for the kind of corruption that happened in my case.

What happens if one of the madam-Raid1 drives holds different data due to one drive having bad sectors? Will I still be able to retrieve the correct file or will the array be unable to decide which file is the correct one and lose it entirely?

If you know definitively that one drive is bad, you can fail that drive out of the array and serve all reads from the good drive (or, more sensibly, replace the bad drive, which will copy the data from the good drive onto the replacement). But if the data on the drives differs due to random bit flips on write (the kind of thing that happened to me and shodanshok) there is no definitive way to choose which of the two is correct without a checksum.

Also, md generally won't notice that two drives in a mirror are out of sync during normal operation — it will direct reads to one disk or the other in whatever way will get the fastest result. There is a 'check' function that will read both sides of a mirror pair and report mismatches, but only if you run it, or if your distribution is set up to run it periodically and report the results.

9
  • 2
    This recent post seems relevant for RAID1 (even though it talks about other situations than bad-RAM data corruption): lwn.net/Articles/819885
    – user1686
    May 17, 2020 at 10:38
  • @user1686, thank you for the link - that explained the situation very well. It also complements this answer, so after all an mdadm Raid is not really enough to ensure data availability and only protects against complete drive failures. May 17, 2020 at 11:44
  • One source of trouble is defective HD firmware or other defects due to misfeatures in other parts. A number of people running ZFS systems had trouble with the SAS to SATA interposers. That type of misbehaviour cant really be fixed with mirroring only. May 17, 2020 at 14:02
  • @Stefan Skoglund, Yes, that was my thought on this as well after reading. When the drives in use have good firmware that detects errors and is very unrelenting (e.g. trustworthy, dedicated NAS-drives, rather than for example 24/7-surveillance drives which might ignore some errors in order to record continuously), then using a simple RAID1-array seems more valid than if you have the feeling that you can't really trust your drives in the first place. May 18, 2020 at 10:59
  • @Prototype700 My drives are NAS drives, and the drives didn't do anything wrong at any point, so that's an odd conclusion to reach :)
    – hobbs
    May 18, 2020 at 21:04
2

I can add that ZFS is insanely robust, mostly thanks to its origins (it was developed by Sun Microsystems back in 2001). The open source version currently available is a fork of one of the last open source versions released by Sun Microsystems around 10 years ago that has been further developed by the open source community after Oracle closed the ZFS source after acquiring Sun Microsystems.

Oracle themselves still also maintain a closed source version of ZFS that's used in their enterprise storage systems.

ZFS has a bit of a learning curve though, as it's quite powerful there's quite a lot of things that can be tweaked. Also it's one of the few storage file systems I've worked on where maintenance is actually easy. I had one case where a pool needed to be migrated from a RAID5 setup to a RAID6 (or more correctly a RAID-Z1 to a RAID-Z2). Normally, an operation like this would mean copying out all the data, reconfiguring the RAID, and copying the data back in. In ZFS, you attach you secondary storage, and the copy the pool out with one command, reconfigure the array as you like, and copy the pool back in with another command.

There are some gotchas though:

  1. To get any benefit from ZFS, you'll need to let ZFS handle the disks themselves. So your drive controller needs to support JBOD, so that ZFS sees the disks directly. All RAID configurations are handled in ZFS, as it uses the parity data for scrubbing and other tasks, it can't have it hidden by a RAID controller.
  2. As others have stated, ECC memory is strongly recommended. ZFS doesn't require it, but it fully expects that anything written to RAM is immutable and won't get corrupted. So if you run it on a system with non-ECC RAM and your memory goes bad, ZFS might actually corrupt your data while it's scrubbing the array (scrubbing means that ZFS reads data from the pool, calculates what it should have read from the parity information saved on other drives, and corrects any errors it finds).
  3. Even though ZFS is very excellent at preventing data loss, its RAID-Z1 still suffers from the same issues as RAID5, aka. that large arrays of large (1TB+) drives can fail completely after a single disk failure when rebuilding the array if the drives URE rate is too high, as just reading back all the data from parity from the rest of the drives while rebuilding mathematically almost guarantees an Unrecoverable Read Error due to the size of the drives. Run RAID6 / RAID-Z2 if you're not an expert on operating storage systems and happen to know what you're doing.

For beginners and home environments I generally recommend FreeNAS, it's very well maintained and simple to set up, which is good for a beginner.

1

How likely am I to even encounter actual data corruption making files unreadable? How?

Obviously, given infinite time you're certain to encounter it.

Realistically though, it's still pretty likely unless you have very expensive enterprise grade hardware, and even then it's not hugely unlikely.

More likely though, you'll end up encountering data corruption that just changes the file contents, but doesn't make them unreadable (unless you've got insane numbers of tiny files, simple statistics means you're more likely to have corruption in file data than in file metadata). When this happens, you can get all kinds of odd behaviors just like if you had bad hardware (though usually it will be more consistent and localized than bad hardware). If you're lucky, it's some non-critical data that gets corrupted, and you can easily fis things. If you're moderately unlucky, you have to rebuild the system from scratch. If you're really unlucky, you just ran into an error that caused you to go bankrupt because it happened to hit critical data in a production system and your service is now down while you rebuild the whole thing from scratch and try to put the database back the way it should be.

Short answer, data corruption is likely enough that even home users should be worrying about it.

Can Ext4 or the system file manager already detect data errors on copy/move operations, making me at least aware of a problem?

Ext4 is notoriously bad on this point. Their default behavior n running into an internal consistency error is to mark the filesystem for check on next remount, and then continue as if nothing is wrong. I've lost whole systems in the past because of this behavior.

More generically, in most cases, the best you can hope for from a filesystem not specifically designed to verify it's data is to remount read-only if it runs into an internal error with it's own data structures or file metadata. The thing is though, unless the filesystem specifically handles verification of it's own internal structures beyond simple stuff like bounds checking, this won't catch everything, things will just go wrong in odd ways.

To get anything more, you need the filesystem to verify it's own internal data structures with checksums, error correcting codes, erasure coding, or some similar approach. Even then, unless it does the same for file data, you're still at non-negligible risk of data loss.

What happens if one of the madam-Raid1 drives holds different data due to one drive having bad sectors? Will I still be able to retrieve the correct file or will the array be unable to decide which file is the correct one and lose it entirely?

It depends on the RAID level, the exact RAID implementation, and whether or not you have it set to auto-recover. Assuming you have auto recovery on:

For RAID1 and RAID10:

  • With hardware RAID and only two replicas, it usually picks the first replica and syncs the array to that.
  • In some hardware RAID systems with more than two replicas, it checks if a majority of the replicas match, and if so overwrites the ones that don't match with that.
  • With software RAID, it usually does the same as with hardware RAID unless it has a clear indication that the discrepancy is a result of a failed write (in which case it picks the copy that it knows was completely written).
  • With BTRFS, it looks at which copy has a correct checksum, and replaces the one that doesn't with that.
  • I believe that ZFS works like BTRFS here.

For RAID4/5/6 and other cases of erasure coding, almost everything behaves the same when it comes to recovery, either data gets rebuilt from the remaining devices if it can be, or the array is effectively lost. ZFS and BTRFS in this case just give you a quicker (in terms of total I/O) way to check if the data is correct or not.

Note that none of these operate on a per-file basis and most don't allow you to easily pick the 'correct' one, they either work completely, fail completely, or alternately return good or bad data for the out of sync region.

0

For completeness, I'd like to mention https://bcachefs.org, which is admittedly not in the kernel yet, but IMHO slated to supplant ZFS and btrfs once it does.

It's based on bcache, which has already been in the kernel for a long time, building file system features with its B-tree system.

The lone developer works on it full time, sponsored via Patreon, and has a strong focus on reliability.

Not for the faint of heart at the moment, but as this comment ages, bcachefs should improve :)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.