ZFS pool corrupted

Question

I am running proxmox (7.1-10) and a number of VMs. I accidentally forced to import from proxmox a ZFS pool that was being used by a VM (truenas 12.0-U7) with:

zpool import -f Pool-1

The command returned blank, and I thought I was ok.

I then rebooted Proxmox, and the pool was not there. I did then:

zpool import

pool: Pool-1
     id: 9292035031829486490
  state: FAULTED
status: The pool metadata is corrupted.
 action: The pool cannot be imported due to damaged devices or data.
    The pool may be active on another system, but can be imported using
    the '-f' flag.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-72
 config:

    Pool-1        FAULTED  corrupted data
      mirror-0    FAULTED  corrupted data
        sdd2      ONLINE
        sdb2      ONLINE
      indirect-1  ONLINE
      indirect-2  ONLINE
      indirect-3  ONLINE

At that time, I had not realised the mistake I was doing and tried to import once again:

zpool import -f Pool-1

internal error: cannot import 'Pool-1': Invalid exchange
Aborted

By then, I realised I was importing the wrong pool. I went back to Truenas, and the pool was offline. I checked whether Truenas could see the pool:

truenas# zpool import

   pool: Pool-1
     id: 9292035031829486490
  state: FAULTED
status: The pool was last accessed by another system.
 action: The pool cannot be imported due to damaged devices or data.
    The pool may be active on another system, but can be imported using
    the '-f' flag.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
 config:

    Pool-1                                          FAULTED  corrupted data
      mirror-0                                      FAULTED  corrupted data
        gptid/f475cf25-9d3a-11eb-a1a4-0cc47a30748c  ONLINE
        gptid/f485d5c5-9d3a-11eb-a1a4-0cc47a30748c  ONLINE
      indirect-1                                    ONLINE
      indirect-2                                    ONLINE
      indirect-3                                    ONLINE

and tried first:

truenas# zpool import Pool-1

cannot import 'Pool-1': pool was previously in use from another system.
Last accessed by proxmox (hostid=2e5301d3) at Tue Jan 25 16:21:58 2022
The pool can be imported, use 'zpool import -f' to import the pool.

and then:

truenas# zpool import -f Pool-1
internal error: cannot import 'Pool-1': Integrity check failed
Abort trap (core dumped)

So, now if I try to force an import from proxmox I get "Invalid exchange" and if I try to force from truenas I get "Integrity check failed".

Does anyone have an idea how to solve this?

Thanks, RG

Nb. The pool was created with encryption.

Answer 1

I proceeded with truenas# zpool import -f -FXn Pool-1

And after 12 hours I got:

Code:
Would be able to return Pool-1 to its state as of Tue Jan 25 16:18:53 2022.
Would discard approximately 3 minutes of transactions.

I then went with zpool import -f -FX Pool-1

And my pool is now safe and sound.

Thanks for all suggestions, RG

Answer 2

Please don't get me wrong, I don't want to be a source of a false hope. Pool is most probably dead, but you have few options:

• try to import the pool in a read-only mode
• try to import the pool with -F recovery mode if your zfs implementation has it
• try to import the pool on a system with the most recent zfs stack that you can possibly get on an original OS the pool was created on, for instance: nightly build, unstable git revision or the most recent available commit in the developer's branch. Since the pool was created on Proxmox, I'd go with it.

Anyway, importing the pool on foreign OSes (like pool created on Solaris - on FreeBSD or Linux, and so on) is generally a bad idea, and should be taken with great caution.