4

I have a Ubuntu server set up with a 14-disk ZFS raidz2 pool.

About 80% of the time, on reboot, I will end up with a degraded pool with two of the disks marked as faulted. The drives that are faulted aren't always the same, but it is always exactly two drives. For example:

$ sudo zpool status
  pool: tank
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: resilvered 4K in 0h0m with 0 errors on Sun Sep 30 23:08:51 2018
config:

        NAME                      STATE     READ WRITE CKSUM
        tank                      DEGRADED     0     0     0
          raidz2-0                DEGRADED     0     0     0
            sde                   ONLINE       0     0     0
            sdc                   ONLINE       0     0     0
            sdd                   ONLINE       0     0     0
            sda                   ONLINE       0     0     0
            sdh                   ONLINE       0     0     0
            11521322863231878081  FAULTED      0     0     0  was /dev/sdf1
            15273938560620494453  FAULTED      0     0     0  was /dev/sdg1
            sdb                   ONLINE       0     0     0
            sdi                   ONLINE       0     0     0
            sdj                   ONLINE       0     0     0
            sdk                   ONLINE       0     0     0
            sdl                   ONLINE       0     0     0
            sdm                   ONLINE       0     0     0
            sdn                   ONLINE       0     0     0

errors: No known data errors

I can export and re-import the pool, and the disks are no longer faulted. For example:

$ sudo zpool export tank
$ sudo zpool import tank
$ sudo zpool status
  pool: tank
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: resilvered 4K in 0h0m with 0 errors on Sun Sep 30 23:08:51 2018
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            sde     ONLINE       0     0     0
            sdc     ONLINE       0     0     0
            sdd     ONLINE       0     0     0
            sda     ONLINE       0     0     0
            sdh     ONLINE       0     0     0
            sdg     ONLINE       0     0     1
            sdf     ONLINE       0     0     0
            sdb     ONLINE       0     0     0
            sdi     ONLINE       0     0     0
            sdj     ONLINE       0     0     0
            sdk     ONLINE       0     0     0
            sdl     ONLINE       0     0     0
            sdm     ONLINE       0     0     0
            sdn     ONLINE       0     0     0

errors: No known data errors

The HBA being used has worked correctly in another server.

Anything else I can try to avoid these faulted drives on reboot? I have another HBA I could swap in.

4

You should not use /dev/sdX names for your pool configuration.

Any change in SCSI enumeration, say inserting a CDROM or a USB drive, could cause the device names to change, resulting in the errors you're experiencing.

You have an option of using the /dev/disk/by-id names.

Do this with a zpool export tank and zpool import -d /dev/disk/by-id tank

  • Thanks, this works. The zpool now seems to be using /dev/disk/by-id/wwn-* device names and imports successfully on reboot. I was expecting device names like ata-* which use the device model and serial number, rather than wwn-* which is the LU WWN Device ID reported by hdparm and smartctl. No worries though, as long as it works. For anybody finding this later, there is a zfs issue that uses this same workaround. unix.stackexchange.com/questions/288599/… – David Chappelle Oct 12 '18 at 2:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.