FreeNAS zfs zraid two failing disks

Question

I have a FreeNAS 8.3 file server with ZFS running four 3TB disks in raidz.

camcontrol devlist:

          at scbus0 target 0 lun 0 (pass0,ada0)
          at scbus1 target 0 lun 0 (pass1,ada1)
          at scbus2 target 0 lun 0 (pass2,ada2)
          at scbus3 target 0 lun 0 (pass3,ada3)
       at scbus6 target 0 lun 0 (pass4,da0)

Last week I've noticed that two of the disks out of four were starting to fail:

freenas smartd[2241]: Device: /dev/ada0, 24 Currently unreadable (pending) sectors
freenas smartd[2241]: Device: /dev/ada0, 24 Offline uncorrectable sectors
freenas smartd[2241]: Device: /dev/ada2, 24 Currently unreadable (pending) sectors
freenas smartd[2241]: Device: /dev/ada2, 24 Offline uncorrectable sectors

zpool status -v didn't show me any errors however. I'm not all that familiar with ZFS, and this was setup by another admin. For some reason I was under the impression that I could simply replace the failed drives one after another. Which is what I proceeded to do.

2014-10-13.17:41:29 zpool offline vol1 gptid/24726389-df9e-11e1-9963-c860009da3f8
2014-10-13.18:19:24 zpool replace vol1 15380758640793782293 gptid/f1a3e8b8-5326-11e4-966d-c860009da3f8
2014-10-13.18:21:28 zpool detach vol1 15380758640793782293

When resilvering process completed I got an error about data corruption in one snapshot. The system is setup to snapshot once per hour, which are saved for two weeks, and once per day for six month.

[root@freenas] ~# zpool status -v
  pool: vol1
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
  scan: resilvered 2.25T in 27h51m with 1 errors on Tue Oct 14 22:10:59 2014
config:

        NAME                                            STATE     READ WRITE CKSUM
        vol1                                            ONLINE       0     0     1
          raidz1-0                                      ONLINE       0     0     2
            gptid/f1a3e8b8-5326-11e4-966d-c860009da3f8  ONLINE       0     0     0
            gptid/24f91374-df9e-11e1-9963-c860009da3f8  ONLINE       0     0     0
            gptid/25865cb9-df9e-11e1-9963-c860009da3f8  ONLINE       0     0     0
            gptid/260cd97a-df9e-11e1-9963-c860009da3f8  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:
 [email protected]:/home/.../some.pdf

Now this file has been deleted long time ago, and so I don't care about it at all, I figured I can just delete the snapshot, but that made things worse:

[root@freenas]~# zfs destroy [email protected]
[root@freenas] ~# zpool status -v
  pool: vol1
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
  scan: resilvered 2.25T in 27h51m with 1 errors on Tue Oct 14 22:10:59 2014
config:

        NAME                                            STATE     READ WRITE CKSUM
        vol1                                            ONLINE       0     0     1
          raidz1-0                                      ONLINE       0     0     2
            gptid/f1a3e8b8-5326-11e4-966d-c860009da3f8  ONLINE       0     0     0
            gptid/24f91374-df9e-11e1-9963-c860009da3f8  ONLINE       0     0     0
            gptid/25865cb9-df9e-11e1-9963-c860009da3f8  ONLINE       0     0     0
            gptid/260cd97a-df9e-11e1-9963-c860009da3f8  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        <0x7c343>:<0x45b6bcd>

So I decided to investigate further, read Oracle docs, and found this: "if two disks in a four-way RAID-Z (raidz1) virtual device are faulted, then neither disk can be replaced because insufficient replicas from which to retrieve data exist. " ada2 is still throwing errors, and it looks like I cannot replace it in the existent pool as it will make the data unavailable?

Does this mean that the only way to recover is to backup existent data, destroy the pool, replace ada2, create a new pool and restore the data on a new pool? Or is there another way of doing this, deleting all snapshots before 20140830 perhaps? Btw latest scrub found another snapshot with corrupt data, same file obviously:

  [email protected]:/home/.../some.pdf

Answer 1

I believe the reason why you are seeing things like

errors: Permanent errors have been detected in the following files:

        <0x7c343>:<0x45b6bcd>

is because the data still exists in a snapshot, and only a snapshot.

Remember that a snapshot in ZFS is only a point-in-time marker; it does not actually copy any blocks, it simply keeps the references to the old blocks alive when the data is replaced (using ZFS' normal copy-on-write behavior). Hence, if there are any snapshots which reference the bad blocks, it will keep showing up during scrubs as an error.

The error only affects that specific file, and your other data is not at any additional risk due to this error.

The "fix" is to destroy each snapshot in turn that contains the affected file. Once all such snapshots have been destroyed, no references to the bad blocks will remain and ZFS will (hopefully) report that there are no errors.

Also, the recommended way to replace a failing but still functional (marginal) device in a ZFS pool is to use zpool replace pool old-dev new-dev with both old-dev and new-dev connected throughout the replacement process. This allows ZFS to use the data on the marginal device where that is possible. Once zpool replace completes, the old device will automatically be removed from the pool and can be physically disconnected. Obviously this requires having the appropriate number of extra interconnects available on the host.