3

I have never encountered a disk failure in my whole life but I had a dozen of bit rot and silent corruptions. 'Normal' file systems and hardware RAID can do absolutely nothing about it. I have been using application-level parity redundancy to protect my most important data, but it's apparently not quite effective.

I know ZFS store checksum of the chunks in software RAID, which offers protection against bit rot and silent corruption, or does it ?. The problem is, I do not want RAID-Z because it does not support expansion, which means I have to purchase disks with larger capacity than all of my data if I want expansion.

So is it possible to use filesystem-level parity without RAID or unequal disk size RAID ? For example by allocation part of the disk space to store the parity of the "chunks" instead of storing the parity of equal sized disks on another disk as in RAID5.

Thanks everyone for answering.The following is the conclusion based on the answers:

No, it's not possible to use ZFS parity without RAID, but it's possible to use mirror copy without RAID.

3
  • 6
    Why is RAID "off the table"?
    – ewwhite
    Mar 18 '19 at 14:25
  • 1
    storing the parity of equal sized disks on another disk as in RAID5 That's not really how RAID5 works. Mar 18 '19 at 15:31
  • What do you mean that RAID can do nothing about it? RAID can repair corrupt data even if a whole disk would go missing from system. Why you think it won't be able to check and repair a few bits?
    – yurtesen
    Feb 21 at 15:57
6

A checksum can only detect corruption. It cannot help you restore the data.

If you aren't willing to mirror your data on two (or more) vdevs, then you can set the copies ZFS property, which will tell ZFS to keep n copies of the data (1, 2 or 3). In case one copy is corrupted, the other can be used to recover the data.

$ zfs get copies srv
NAME  PROPERTY  VALUE   SOURCE
srv   copies    1       default

$ zfs set copies=2 srv

Keep in mind that this does not protect you from an actual disk failure (which can happen even if you have never experienced it before). This is not a backup. Make appropriate backups of your data.

5
  • 1
    Nor does having multiple copies stored in the filesystem protect you if you delete the file yourself. Mar 18 '19 at 15:30
  • 1
    @AndrewHenle This is achievable via snapshots (and backups)
    – shodanshok
    Mar 18 '19 at 15:39
  • "A checksum can only detect corruption. It cannot help you restore the data" is only true for a single simple checksum. More complex systems consisting of multiple checksums in the simplest case can be used to correct data. Look up Error Correcting Codes (ECC). Consider a table of bits, where every row and column is checksummed, Then when 1 bit is in error that row and column checksum will fail. The row / column intersection of failed checksums will tell you which bit to flip to correct the problem. It gets tricky to guard against multiple errors, particularly in the same row or column. Apr 10 '19 at 5:21
  • 1
    @BeowulfNode42 This question and answer is specifically about ZFS. All of that is true, but completely irrelevant. Apr 10 '19 at 14:45
  • 1
    ECC checksums are also extremely weak and prone to hash collisions that render them insensible to the fact that the data is corrupt at all. Modern hard drives almost univerally implement ECC checksumming at the hardware block level, which does not prevent them from cheerfully returning the occasional corrupt block anyway.
    – Jim Salter
    May 5 '19 at 0:27
7

I don't think this is a good use case for ZFS. It's like you have a desired solution and are trying to make this fit.

Yes, there's the ZFS copies= parameter, which will endeavor to store multiple copies of files on different sectors of the disks, but it's not a substitute for true redundancy.

Please see: https://jrs-s.net/2016/05/02/zfs-copies-equals-n/

When you post a question and say such firm things like "RAID is off the table", it may be helpful to explain why you have those constraints so that we can help answer the real question.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.