2

I have a ZFS pool that had two drives go bad on the same day. Now I'm left in a state where it is in a continuous resliver process that never seems to finish. In the meantime, I'm just trying to copy some of the data off to another file server and the volume is almost unusable (like 500kBps disk access). The server reboots when it gets about 70% reslivered then it starts all over again.

I'm looking for two pieces of advice: 1) Can I stop the resliver temporarily so I can copy the data I need off the drive (it is about 1TB that I need to copy in total) 2) Is this array salvageable? I'm not sure, but it looks like mirror-1 has issues with both its mirrored drives, and from my understanding that is not something that can normally be recovered from.

pool: primary_vol

state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Sat Dec 28 20:01:33 2019 105G scanned out of 7.28T at 15.7M/s, 133h2m to go 26.2G resilvered, 1.41% done config:

    NAME           STATE     READ WRITE CKSUM
    primary_vol    DEGRADED   215     0     0
      mirror-0     ONLINE       0     0     0
        c0t12d1    ONLINE       0     0     0
        c0t13d1    ONLINE       0     0     0
      mirror-1     DEGRADED   215     0    35
        spare-0    DEGRADED   430     0     0
          c0t15d1  FAULTED      0     0     0  too many errors
          c0t21d1  ONLINE       0     0   430  (resilvering)
        c0t18d1    DEGRADED   215     0    59  too many errors  (resilvering)
      mirror-2     ONLINE       0     0     0
        c0t19d1    ONLINE       0     0     0
        c0t20d1    ONLINE       0     0     0
      mirror-3     DEGRADED     0     0     0
        c0t24d1    ONLINE       0     0     0
        c0t22d1    UNAVAIL      0     0     0  cannot open
    logs
      c0t16d1      ONLINE       0     0     0
    spares
      c0t21d1      INUSE     currently in use

errors: 184 data errors, use '-v' for a list

zpool status -v shows the following errors. These are all files I don't care about. Would deleting them help at all?

errors: Permanent errors have been detected in the following files:

/volumes/primary_vol/vmware/mgmt.cloud.abraxis.net/mgmt.cloud.abraxis.n .et-flat.vmdk
    /volumes/primary_vol/vmware/guzman/guzman-flat.vmdk
    /volumes/primary_vol/vmware/vangas_5/vangas_5-flat.vmdk
    /volumes/primary_vol/vmware/kelly_01/kelly_01-flat.vmdk
    /volumes/primary_vol/vmware/allen_01/allen_01-flat.vmdk

zpool get version primary_vol
NAME         PROPERTY  VALUE    SOURCE
primary_vol  version   28       default
  • What’s the OS and ZFS version? There are a few ways to correct this. I have some tooling one ZFS for Linux that cabin assist. – ewwhite Dec 29 '19 at 3:17
  • Sorry, I see it’s Solaris. Do you have version info? – ewwhite Dec 29 '19 at 3:22
  • I'm running nextenta Community v3 something, so running OpenSolaris. ZFS 28 – John P Dec 29 '19 at 4:53
  • Have you contacted Nexenta? You can remove the files showing errors. Those are VMware VMDKs, so be careful. That may help your resilver. Do you know what type of drives these were? – ewwhite Dec 29 '19 at 5:35
  • I'll try removing the files. They are VMs that were turned off long ago. I don't have a support contract with Nextenta and since I'm running an old community edition, I doubt they would be much help. To my question #2 - is mirror-1 hosed permanently? – John P Dec 29 '19 at 5:53
1

You have drive failures, but one disk was or is being replaced with a spare. The other disk is part of a mirror. You need two replacement drives.

This is old Nexenta, so many of the tools and tips I have for ZFS on Linux May not apply.

I asked about the types of drives in use. Do you have make/model/capacity? How about general hardware information?

  • 10 x 2TB drives (Hitachi SATA). LSI RAID card setup as JBOD - ZFS handles the actual RAID. RAID 10 (or mirrored in ZFS parlance). The server is just a generic Xeon 2U. Unfortunately I don't have a way to identify the drives - nothing I've tried so far allowed me to associate a serial number to the "c0t22d1" named assigned it by Nextenta. – John P Jan 1 at 0:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.