Background: small personal server, email, web, shell, etc, for family and friends. SunOS 5.11, snv_113 from November 2008. Server built in 2009. I believe that was either opensolaris or Solaris early access release. AMD 64 bit processor, 4gb ram.

Root zpool three-way mirror, originally comprised of three laptop-size 320gb spinning disks. After three years, each of the spinning disks died off, one by one; each one replaced under warranty from manufacturer. In the last few months, another disk went stupid again. For the time being, I decided to just run a two way mirror. In the last week, a permanent error cropped up, with three files listed. After a scrub, those errors went away except for one metadata error. As the second disk began faulting as well, I threw in a spare desktop drive and resilvered to that. The same checksum and metadata error persisted. In desparation, I purchased a couple of SSDs (I've come to really hate spinning disks). I've added one to the pool as a third, and of course during resilver, I remain with the following:

root-klaatu /root% zpool status -v
  pool: rpool
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
 scrub: resilver in progress for 1h47m, 84.53% done, 0h19m to go


    rpool       ONLINE       0     0     1
      mirror    ONLINE       0     0     6
        c1d0s0  ONLINE       0     0     6
        c1d1s0  ONLINE       0     0     6  55.0G resilvered
        c0d1s0  ONLINE       0     0     6  55.3G resilvered


Permanent errors have been detected in the following files:

By the time it finishes there will be ten checksum errors per disk reported, and two on the rpool itself. A 'zpool clear' works "great" for clearing the checksum error count, but the errored metadata file persists, and each resilver/scrub just brings back the same checksum errors. Note that I have deleted all snapshots, as I've read that sometimes that's where the errored data lives - but no joy.

I have backups - but they are via CrashPlan, and in the cloud - so restoring the entire thing would take days over the limited connection I maintain for the server, which means days of the server being unavailable, which simply isn't tenable.

Soooo - all that said. What is my option for recovery other than restoring from backup (since the metadata 'file' is obviously something I can't delete).

If I were to create a new zfs pool on a new disk - would doing a zfs send/receive 'clear' the error? Or would it be better to set up the new pool, then rsync the old pool data to it, then rename the new pool back to the original, install the boot blocks on the mirror disks, then try booting from that? I've read that there are cached bits of data related to the actual pool name that may interfere with actually booting.

I know that the 'logical' thing to do would be to just build a new, fresh server on a recent OS like OmniOS or OpenIndiana or whatnot - but with all of the custom compiled code on this server (which was originally a Sparc 20 back in the early 2000's), I presume that rsyncing everything over would not work well.

Thanks for any help.

Oh - I should add - the server is running just fine. No crashes, no lockups, nothing.

  • I hope you get a fix for this, but let me note that backups from which you can't restore in a business-acceptable timeframe aren't backups, they're a security blankie. Once this is sorted, you need to review your policy urgently. – MadHatter Aug 8 '15 at 7:14
  • Forgive me, but what part of the first four words of my post were unclear? – anastrophe Aug 8 '15 at 20:26
  • 1
    None of them, though three of them make the post off-topic for ServerFault, which is about managing information technology systems in a business environment - so I'm not sure how much attention I'd draw to them. In any case, unless you don't value this data, whether or not you need better backups is entirely a function of whether your existing strategy does what you want - and yours, by your own admission, doesn't. – MadHatter Aug 9 '15 at 3:39

I was able to 'recover' from this incident by zfs 'send'ing and receiving the corrupt pool to a new pool.

# create new single disk pool (or a mirror in advance if you wish)
zpool create -f tpool c0d0s0

# make a baseline snapshot of the old pool
zfs snapshot -r rpool@now

# zfs send it to the new tpool
zfs send -vR rpool@now | zfs receive -Fduv tpool

# note that this will reset tpool's mountpoint to rpool - so be sure to update it
zfs set mountpoint=/tpool tpool

# go single user; how depends on your system
# edit:

# (Note that this file is supposed to be maintained by bootadm; under the circumstances, I chose to just go direct)

#duplicate first bootset, edit the copy, change findroot from
findroot (pool_rpool,0,a)
# to
findroot (pool_tpool,0,a)

# make a second snapshot now that in single user
zfs snapshot -r rpool@now2

# send the incremental snap to the new tpool
zfs send -vR -i rpool@now rpool@now2 | zfs receive -Fduv tpool

# mount tpool - bear in mind again, mountpoint needs to be updated
zfs mount=/tpool tpool

# rm the 'bootsign' file and replace with new:
rm /tpool/boot/grub/bootsign/pool_rpool
touch /tpool/boot/grub/bootsign/pool_tpool

# formalize where to boot from
zpool set bootfs=tpool/ROOT/snv_113 tpool

# add the boot blocks
installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c0d0s0

# poweroff, detach rpool disks - or leave them for further investigation
# if going to mirror, physically attach the second disk
# boot up
# turn tpool into a mirror
zpool attach tpool c0d0s0 c1d0s0

# done

Note that if everything is satisfactory, you can edit the grub menu to either move the new tpool boot entry to be the first, or change the 'default' declaration to point to whichever one it is in the list (or, if there are no other boot declarations, then just delete the one for rpool).

Also - while I referred to probably more than a hundred different sites and webpages while trying to solve this, the above 'recipe' is largely derived from How to shrink a mirrored ZFS rpool, by Joe Mocker.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.