Background: small personal server, email, web, shell, etc, for family and friends. SunOS 5.11, snv_113 from November 2008. Server built in 2009. I believe that was either opensolaris or Solaris early access release. AMD 64 bit processor, 4gb ram.
Root zpool three-way mirror, originally comprised of three laptop-size 320gb spinning disks. After three years, each of the spinning disks died off, one by one; each one replaced under warranty from manufacturer. In the last few months, another disk went stupid again. For the time being, I decided to just run a two way mirror. In the last week, a permanent error cropped up, with three files listed. After a scrub, those errors went away except for one metadata error. As the second disk began faulting as well, I threw in a spare desktop drive and resilvered to that. The same checksum and metadata error persisted. In desparation, I purchased a couple of SSDs (I've come to really hate spinning disks). I've added one to the pool as a third, and of course during resilver, I remain with the following:
root-klaatu /root% zpool status -v pool: rpool state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: resilver in progress for 1h47m, 84.53% done, 0h19m to go
NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 1 mirror ONLINE 0 0 6 c1d0s0 ONLINE 0 0 6 c1d1s0 ONLINE 0 0 6 55.0G resilvered c0d1s0 ONLINE 0 0 6 55.3G resilvered
Permanent errors have been detected in the following files: <metadata>:<0x172>
By the time it finishes there will be ten checksum errors per disk reported, and two on the rpool itself. A 'zpool clear' works "great" for clearing the checksum error count, but the errored metadata file persists, and each resilver/scrub just brings back the same checksum errors. Note that I have deleted all snapshots, as I've read that sometimes that's where the errored data lives - but no joy.
I have backups - but they are via CrashPlan, and in the cloud - so restoring the entire thing would take days over the limited connection I maintain for the server, which means days of the server being unavailable, which simply isn't tenable.
Soooo - all that said. What is my option for recovery other than restoring from backup (since the metadata 'file' is obviously something I can't delete).
If I were to create a new zfs pool on a new disk - would doing a zfs send/receive 'clear' the error? Or would it be better to set up the new pool, then rsync the old pool data to it, then rename the new pool back to the original, install the boot blocks on the mirror disks, then try booting from that? I've read that there are cached bits of data related to the actual pool name that may interfere with actually booting.
I know that the 'logical' thing to do would be to just build a new, fresh server on a recent OS like OmniOS or OpenIndiana or whatnot - but with all of the custom compiled code on this server (which was originally a Sparc 20 back in the early 2000's), I presume that rsyncing everything over would not work well.
Thanks for any help.
Oh - I should add - the server is running just fine. No crashes, no lockups, nothing.