I am trying to understand ZFS' behaviour under a specific condition, but the documentation is not very explicit about this so I'm left guessing.

Suppose we have a zpool with redundancy. Take the following sequence of events:

  1. A problem arises in the connection between device D and the server. This causes a large number of failures and ZFS therefore faults the device, putting the pool in degraded state.

  2. While the pool is in degraded state, the pool is mutated (data is written and/or changed.)

  3. The connectivity issue is physically repaired such that device D is reliable again.

  4. Knowing that most data on D is valid, and not wanting to stress the pool with a resilver needlessly, the admin instead runs zpool clear pool D. This is indicated by Oracle's documentation as the appropriate action where the fault was due to a transient problem that has been corrected.

I've read that zpool clear only clears the error counter, and restores the device to online status. However, this is a bit troubling, because if that's all it does, it will leave the pool in an inconsistent state!

This is because mutations in step 2 will not have been successfully written to D. Instead, D will reflect the state of the pool prior to the connectivity failure. This is of course not the normative state for a zpool and could lead to hard data loss upon failure of another device - however, the pool status will not reflect this issue!

I would at least assume based on ZFS' robust integrity mechanisms that an attempt to read the mutated data from D would catch the mistakes and repair them. However, this raises two problems:

  1. Reads are not guaranteed to hit all mutations unless a scrub is done; and

  2. Once ZFS does hit the mutated data, it (I'm guessing) might fault the drive again because it would appear to ZFS to be corrupting data, since it doesn't remember the previous write failures.

Theoretically, ZFS could circumvent this problem by keeping track of mutations that occur during a degraded state, and writing them back to D when it's cleared. For some reason I suspect that's not what happens, though.

I'm hoping someone with intimate knowledge of ZFS can shed some light on this aspect.

  • You're basically asking what happens if a disk in a pool is disconnected, then reconnected, but you don't allow the array to repair itself (resilver)? Why? What is the use-case? Fix the unstable drive connection. – ewwhite Jun 19 '13 at 12:11
  • @ewwhite I'm not proposing to leave an unstable connection - I'm concerned with what happens once it's fixed. Per docs, "If the device errors are deemed transient, in that they are unlikely to affect the future health of the device, they can be safely cleared to indicate that no fatal error occurred." But docs also imply that this only resets error counters. They don't suggest ZFS makes any effort here to write in missed data, and if it doesn't, simply following their advice will leave the pool in an inconsistent state, risking data loss. – Kevin Jun 19 '13 at 19:14

"Theoretically, ZFS could circumvent this problem by keeping track of mutations that occur during a degraded state, and writing them back to D when it's cleared. For some reason I suspect that's not what happens, though."

Actually, this is almost exactly what it can do in this situation. See, every time the disk in a ZFS pool is written to, the current global pool transaction id is written to the disk. So say, for instance, that you have the scenario you explain occur, and the total time between the connection loss and recovery is less than 127 * txg_timeout (and that's making a lot of gross assumptions about load on the pool and a few other things, but say half that for typical safety's sake, so if txg_timeout is 10 seconds, then 600 seconds or 10 minutes is a reasonable time to expect this to still work).

At the moment before disconnection, the pool was able to successfully write writes related to transaction id 20192. Time passes, and the disk comes back. At the time the disk is once again available, the pool has had a number of transaction groups go through, and is at transaction id 20209. At this point, there is still every possibility ZFS can do what is called a 'quick resilver', where it resilvers the disk, but ONLY for transaction id's 20193 through 20209, as opposed to a full resilver of the drive. This quickly and efficiently gets the disk back up in spec with the rest of the pool.

However, the method to kick off that activity is not 'zpool clear'. If everything works as it should, the resilver should have been kicked in automatically the moment the disk became healthy again. In fact, it may have been so fast, you never saw it. In which case, 'zpool clear' would be the proper activity to clear up the still-visible error count that would have appeared when the device disappeared in the first place. Depending on the version of zfs you're using, what OS it is on, in what manner the device is being listed by zfs at the moment and how long it has been in that state, the 'proper' way to fix this varies. It could actually be 'zpool clear' (clearing up the errors, and the next access of the drive should notice the out of sync txg id and kick in the resilver) or you might need to use 'zpool online' or 'zpool replace'.

What I'm used to seeing, when all this works properly, is the disk disappearing and the drive going into a state of OFFLINE or DEGRADED or FAULTED or UNAVAIL or REMOVED. Then, when the drive becomes accessible again at an OS level, FMA and other OS mechanisms kick in and ZFS becomes aware the disk has returned, and there's a quick resilver and the device appears in zpool status as ONLINE again, but may still have an error count associated with it. The key is it is in ONLINE status, which would indicate automatic repair (resilver) success. You can test it on any drive by pulling it out, waiting a few seconds and checking 'zpool status', and then plugging the disk back in and checking 'zpool status' again and seeing what happens. ZFS isn't the only moving piece here - ZFS actually relies in large part on other OS mechanics to inform it of the disk's status, and if those mechanics fail you'll get different symptoms than if they succeed.

In either event, either the quick resilver is able to be run and succeeds, or it is not possible or fails. If the latter, the disk will have to complete a full resilver before returning to duty, so your two problems listed at the bottom of your post shouldn't usually be possible unless administrative override has allowed a disk with a mismatched txgid to re-enter a pool without any form of correction for that disparity (should not usually be possible). IF that were to happen, I would suspect the next access to the drive would either result in a kick off that quick resilver (and succeed, or fail and knock the disk to a full resilver) or it would end up kicking the disk out -- or possibly panicking, due to the txgid disparity. In any of those events, what would not happen is data loss or a return of incorrect data to a request.

  • Thank you for the good explanation. I gather the quick resilver works when the drive returns to ONLINE automatically through OS mechanisms. However I had a case where a connection failed and required manual intervention to restore. The disk did not return to ONLINE; it stayed FAULTED, but a zfs clear brought it back ONLINE with no further complaints. The downtime was way over 10 minutes, although at a time of rather low load – Kevin Jul 30 '13 at 4:37
  • 'Rather low load' and 'zero load' are two very different scenarios for this circumstance. 'Rather low load' would still cause transaction commits every X seconds, even if it was a single byte of change. Only zero load whatsoever would postpone txg commits (take that statement with a grain of salt; I've never actually tested to see if that's true). – Nex7 Aug 6 '13 at 21:15
  • The issue, as I tried badly to explain, is that this is a very 'perfect world' result. It'll happen, but not every time. The reason is ZFS isn't the only thing involved here. FMA, drivers, firmware, lots of other pieces have to behave properly. IF every piece involved properly notices and flags the disk as gone AND as come back, everything will happen without your intervention. That's a very big IF. :) – Nex7 Aug 6 '13 at 21:15

Note that:

  1. Every chunk of data has fair checksum on ZFS. So ZFS know which drive holds correct data in redundant setup when failure. Running zpool scrub ZFSPOOL will repair data or spread data to all running drives for RADZ.
  2. ZFS employs Reed-Solomon's error correction which is best for bursts of errors. Missing drive is such burst of errors, which R-S can correct.

I got many DMA errors on drives, when air condition issue in datacenter and ZFS was able to fix that mess. And it was just simple mirror.

I do remember promo video issued by SUN when they introduced ZFS... they made RAIDZ on USB flash drives deployed to 8 port USB hub and then randomly changed position in hub for few of them while doing IO on that pool observing no outage.

  • Interesting demo, which doesn't surprise me. ZFS is indeed quite good at detecting and correcting soft errors, which is why I prefer it. I had figured however that after simply clearing a fault, redundancy is compromised on the mutated (new) data, until a scrub is done, which furthermore has to be invoked manually as you suggest. – Kevin Jun 26 '13 at 23:22
  • From man zpool: Scrubbing and resilvering are very similar operations. The difference is that resilvering only examines data that ZFS knows to be out of date (for example, when attaching a new device to a mirror or replacing an existing device), whereas scrubbing examines all data to discover silent errors due to hardware faults or disk failure. Btw to have more comfort, choose NexentaStor: nexentastor.org – nudzo Jun 27 '13 at 9:45
  • As best as I can tell, zpool clear neither scrubs nor resilvers. It just continues as if everything is up-to-date even though a brief outage of the disk means some data might be out of date. You could manually perform a scrub afterwards, but, as you mention, this checks over all data, not just the out-of-date data. – Kevin Jun 28 '13 at 7:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.