cannot import a zpool volume : I/O error

Question

this is on nexenta open storage. I have a zpool that is in UNAVAIL state because of some disks.

# zpool import -F
  pool: delta
    id: XYAZA
 state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
        devices and try again.
config:

        delta                        UNAVAIL  insufficient replicas
          raidz2-0                   UNAVAIL  insufficient replicas
            c0t5000C500573978EBd0    ONLINE
            c0t5000C50057397933d0    ONLINE
            c0t5000C50057397C47d0    UNAVAIL  cannot open
            c0t5000C5005739997Fd0    UNAVAIL  cannot open
            c0t5000C50057399F87d0    ONLINE
            c0t5000C5005739A90Bd0    ONLINE
            c0t5000C5005739ABC7d0    UNAVAIL  cannot open
            c0t5000C5005739AC0Fd0    ONLINE
          raidz2-1                   DEGRADED
            c0t5000C50057398657d0    ONLINE
            c0t5000C50057398E03d0    ONLINE
            c0t5000C50057398EF7d0    ONLINE
            c0t5000C50057398FAFd0    ONLINE
            c0t5000C5005739A39Fd0    UNAVAIL  cannot open
            c0t5000C5005739AAFFd0    ONLINE
            spare-6                  DEGRADED
              c0t5000C5005739AC3Fd0  UNAVAIL  cannot open
              c0t5000C50057398EDBd0  ONLINE
            c0t5000C5005739AE9Bd0    ONLINE


~# zpool import -FXf delta
cannot import 'delta': I/O error
        Destroy and re-create the pool from
        a backup source.

1. How can I get this zpool import work fine?
2. Are the disks that are pointed out 'UNAVAIL', how can I get them repair/replaced?

Update: 7 Sep, 2018

After finding the disks to slot mapping on Nexenta, reseating the disks in delta volume, I could get one disk ONLINE in raidz2-1 group but the 3 disks in raidz2-0 group still continue to show as UNAVAIL.

# zpool import -F
  pool: delta
    id: XYAZA
 state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
        devices and try again.
config:

        delta                        UNAVAIL  insufficient replicas
          raidz2-0                   UNAVAIL  insufficient replicas
            c0t5000C500573978EBd0    ONLINE
            c0t5000C50057397933d0    ONLINE
            c0t5000C50057397C47d0    UNAVAIL  cannot open
            c0t5000C5005739997Fd0    UNAVAIL  cannot open
            c0t5000C50057399F87d0    ONLINE
            c0t5000C5005739A90Bd0    ONLINE
            c0t5000C5005739ABC7d0    UNAVAIL  cannot open
            c0t5000C5005739AC0Fd0    ONLINE
          raidz2-1                   DEGRADED
            c0t5000C50057398657d0    ONLINE
            c0t5000C50057398E03d0    ONLINE
            c0t5000C50057398EF7d0    ONLINE
            c0t5000C50057398FAFd0    ONLINE
            c0t5000C5005739A39Fd0    ONLINE
            c0t5000C5005739AAFFd0    ONLINE
            spare-6                  DEGRADED
              c0t5000C5005739AC3Fd0  UNAVAIL  cannot open
              c0t5000C50057398EDBd0  ONLINE
            c0t5000C5005739AE9Bd0    ONLINE

I found a single spare disk for another online pool on the appliance. Now the question is 1. Should I detach/attach the spare to raidz2-1 group which has only one disk UNAVAIL under sub category of spare-6 ? or 2. Should I detach/attach the spare to raidz2-0 group which still has 3 disks UNAVAIL ?

Given that this pool is offline (after export) and is not importable now.

I looked at some history on how this pool was created, it was created with the below command in the past

zpool create -m /volumes/delta -o autoreplace=on delta raidz2 c0t5000C500573978EBd0 c0t5000C50057397933d0 c0t5000C50057397C47d0 c0t5000C5005739997Fd0 c0t5000C50057399F87d0 c0t5000C5005739A90Bd0 c0t5000C5005739ABC7d0 c0t5000C5005739AC0Fd0 raidz2 c0t5000C50057398657d0 c0t5000C50057398E03d0 c0t5000C50057398EF7d0 c0t5000C50057398FAFd0 c0t5000C5005739A39Fd0 c0t5000C5005739AAFFd0 c0t5000C5005739AC3Fd0 c0t5000C5005739AE9Bd0

Does that indicate raidz2-0 and raidz2-1 are two RAIDZ mirror pools?

If yes, should I prioritize fixing raidz2-1 group of disks to become online over raidz2-0 (which has 3 disk failures)?

Answer 1

1. Should I detach/attach the spare to raidz2-1 group which has only one disk UNAVAIL under sub category of spare-6 ?

Your spare disk has already been automatically attached to raidz2-1 and has been resilvered. Normally, you would now detach c0t5000C5005739AC3Fd0 and remove it form the system because it has failed once (or repurpose it on another system and hope for the best, if you don't have any resources to spare).

But that would not help you, as raidz2-0 is still down. If any vdev in a zpool fails, your whole pool and all data on it is lost. Each vdev will fail if the configured parity cannot be upheld, because enough disks have failed inside it. In your case, each Z2 vdev can sustain the loss of two disks, but one of them has lost three.

2. Should I detach/attach the spare to raidz2-0 group which still has 3 disks UNAVAIL ?

First, you have no real spare now. And even if you removed a working disk from raidz2-1 (not recommended!), you would still gain nothing until you fix your failed disks in raidz2-0. The parity data has to come from somewhere, and in this case it has to come from your failed disks. If you don't get a t least one of them to work, your best bet would be to restore from backup.

I looked at some history on how this pool was created, it was created with the below command in the past zpool create -m /volumes/delta [...] Does that indicate raidz2-0 and raidz2-1 are two RAIDZ mirror pools?

No, that means your pool consists of two striped vdevs, each configured with raidz2 redundancy and 8 disks. You can lose 2 disks in each vdev at most. If you lose 3 in any of them, it's game over.

In ZFS, all redundancy is handled at the vdev level. The pool is always striped over all existing vdevs. If any vdev fails, the pool fails. Mirroring is a type of redundancy for a vdev and is incompatible with raidz1/z2/z3 (you can mix and match vdevs with different redundancy types, but the weakest one will define the strength of your pool and the performance will suffer, so this is not recommended).

If yes, should I prioritize fixing raidz2-1 group of disks to become online over raidz2-0 (which has 3 disk failures)?

No, you must first fix raidz2-0.

---

All of them belong to Seagate, there is a mix of SEAGATE, Rev. A001 and SEAGATE, Rev. 0002, size 1.82TB

This doesn't help you now, but maybe in the future: It is advisable to mix and match manufacturers and models and even manufacture dates, to even out cases of "a bad batch of drives". For example, if you bought 4 disks of each of the 4 major manufacturers (Seagate, Western Digital, Hitachi, Toshiba) and split them 2-2 on your vdevs, even if all disks of one manufacturer failed at the exact same time, you would still have a working pool.