Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 8 down vote favorite
4

With zfs, if you have copies=2 and then you lose a drive containing some of those copies, how do you tell the system that it should make a new copy of the data blocks for the affected files? Or does zfs just start adding data blocks for the extra copies as soon as it finds out about bad data blocks?

Will scrub do this?

(v0.6.0.56-rc8, ZFS pool version 28, ZFS filesystem version 5, Ubuntu 11.10)

share|improve this question

2 Answers

up vote 5 down vote accepted

"copies=2" (or 3) is more designed to be used with pools with no redundancy (single disk or stripes). The goal is to be able to recover minor disk corruption, not a whole device failure. In the latter case, the pool is unmountable so no ditto blocks restoration can occur.

If you have redundancy (mirroring/raidz/raidz2/raidz3), the ditto blocks are not different than other ones and scrubbing/resilvering will recreate them.

share|improve this answer
This directly conflicts with what @Redmumba says - and Redmumba provides links to code. Can you cite some sources for what you're saying? In particular, I'd love to see good citations for why you think copies=N isn't going to cope with whole device failure - that doesn't match with anything I've read. – James Moore Apr 11 '12 at 3:58
@James Moore After a whole device failure, no ditto blocks will be written on that disk. There is no redundancy at the pool level so there is no way to replace the faulty disk by a new one. The only method to properly recover that situation would be to do a full backup of the pool, recreate it with healthy devices, and restore from backup while making sure no unintentional reboot occur before the first backup is done. Otherwise the pool might not be importable and its data lost. This is quite a burden compared to redundant pools where recovering a bad disk is done on-line and survives reboots. – jlliagre Apr 11 '12 at 22:07
"There is no redundancy at the pool level so there is no way to replace the faulty disk by a new one." This doesn't seem relevant. Why do I care about redundancy at the pool level? copies=3 gives me redundancy at the file level. It sure seems like replacing the failing disk + scrub gets back to a good state. – James Moore Apr 11 '12 at 22:54
1  
Here is a reference: docs.oracle.com/cd/E19082-01/817-2271/gbbvf/… For a device to be replaced, the pool must be in the ONLINE state. The device must be part of a redundant configuration, or it must be healthy (in the ONLINE state). I assume copies=2 or 3 is not considered to be a redundant configuration. – jlliagre Apr 12 '12 at 12:17
1  
One thing to keep in mind, though, is that if you originally had copies=1 and you've upped it to copies=2, then you'll probably want to resilver/rescrub afterwards--which will create these instances. But @jilliagre is correct: ditto blocks do not constitute a redundant configuration. There is NO guarantee that the blocks are set on another device, even if you have multiple devices in a pool. – Andrew M. Apr 12 '12 at 16:44
show 6 more comments
up vote 8 down vote

I found this question really intriguing, and after spending an hour pouring over documentation, I dived into the code. Here's what I found.

First, some terminology. Ditto blocks (which are what these copies are, as opposed to mirrors) are automatically created on a write but may or may not be in the same virtual device (vdev) as the original copy. On the other hand, mirrored blocks are always reflected onto another virtual device.

However, the code refers to both types of blocks as children. You'll see here that ditto blocks are just children with io_vd == NULL (this is in the write function). For a mirrored block, io_vd would be set to the corresponding virtual device (your second disk, for example).

With that in mind, when it gets to the read portion, it treats all children (be they mirror or ditto blocks) as potentially unsafe if it doesn't contain the expected good_copies, and rewrites them as needed. So it sounds like the answer to your question is--yes, it will rewrite them when you have at least one good copy, and either of the following:

  • Unexpected errors when you tried to read the data,
  • You are resilvering, or
  • You are scrubbing.

Phew! Maybe someone can point out flaws, but I enjoyed learning about ZFS through this little exercise, and I hope this helps!

share|improve this answer
The problem is in @jlliagre's answer - the pool is dead if it loses any device. The fact that the pool still has enough ditto blocks doesn't seem to matter. Any way around that? – James Moore Apr 12 '12 at 16:09
2  
@JamesMoore you can force the array online in a degraded state if you have the first 1MB of the device that failed. Presumably you just need the metadata from the failed device. I've tested this with a jbod-style zpool and it works : recovering raidz broken labels‌​. I did an md5sum before and after I broke the zpool, and only the copies=1 filesystem was broken after the import. The copies=2 and copies=3 filesystems matched up perfectly. – Jodie C Jun 27 '12 at 3:12

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.