1

(Note I will amend with zpool status for clarity, when I return to the server)

I have a 6x2TB disk raidz2 ZFS pool which I am hosting in Solaris 11

My original M1015 (flashed to IT mode) installed 2 years ago when I built this server failed and was no longer recognized on the PCI-E bus. I replaced it with another M1015 yesterday (flashed to IT mode) and Solaris found all the disks again.

The ZFS pool however was put into SUSPENDED mode (probably due to the previous M1015 dying while running and all the disks vanishing before) and I see resilvering occurring on 2 disks (??), with all disks listed as unavailable.

I have no doubt there are some errors in the pool. But I have cleared (fmadm repaired, and zpool clear) the faults in hopes that the pool could be remounted in degraded state. However upon reboots the pool first comes up as DEGRADED (some disks show as unavailable, some as degraded) , then immediate transitions to SUSPENDED with all disks showing as unavailable and resilvering starts.

The resilvering speed starts at about 100MBps and rapidly ramps down to 50kbps or less. This equates to several hundred hours of expected resilvering time. What's more is that iostat shows ZERO transactions occurring on any of the disks in the pool. It seems that all transactions occur in a burst shortly after a reboot, as I can see about +1GB scanned progress after sequential reboots.

I have no way to offline any of the disks or export the pool while it is in suspended state (also, I have no certain idea why it is entering suspended state when "fmadm faulty"entries are all reported as repaired)

Where have I gone wrong in replacing the SAS controller and how do I recover?


$ zpool status
  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          c8t0d0s1  ONLINE       0     0     0

errors: No known data errors

  pool: tank
 state: SUSPENDED
status: One or more devices is currently being resilvered.  The pool will
        continue to function in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Apr 16 19:37:54 2015
    14.7G scanned out of 8.71T at 127K/s, (scan is slow, no estimated time)
    1.23G resilvered, 0.17% done
config:

        NAME                       STATE     READ WRITE CKSUM
        tank                       UNAVAIL      0     0     0
          raidz2-0                 UNAVAIL      0     0     0
            c0t5000C5005E169C55d0  UNAVAIL      0     0     0
            c0t5000C5005C08BE07d0  UNAVAIL      0     0     0
            c0t5000C5005C07780Ad0  UNAVAIL      0     0     0
            c0t5000C5005E21AE92d0  UNAVAIL      0     0     0  (resilvering)
            c0t5000C5005E0C5056d0  UNAVAIL      0     0     0
            c0t5000C5005C04F982d0  UNAVAIL      0     0     0  (resilvering)

device details:

        c0t5000C5005E169C55d0    UNAVAIL          experienced I/O failures
        status: FMA has faulted this device.
        action: Run 'fmadm faulty' for more information. Clear the errors
                using 'fmadm repaired'.

        c0t5000C5005C08BE07d0    UNAVAIL          experienced I/O failures
        status: FMA has degraded this device.
        action: Run 'fmadm faulty' for more information. Clear the errors
                using 'fmadm repaired'.
           see: http://support.oracle.com/msg/ZFS-8000-GH for recovery

        c0t5000C5005C07780Ad0    UNAVAIL          experienced I/O failures
        status: FMA has faulted this device.
        action: Run 'fmadm faulty' for more information. Clear the errors
                using 'fmadm repaired'.

        c0t5000C5005E21AE92d0    UNAVAIL          experienced I/O failures
        status: FMA has degraded this device.
        action: Run 'fmadm faulty' for more information. Clear the errors
                using 'fmadm repaired'.

        c0t5000C5005E0C5056d0    UNAVAIL          experienced I/O failures
        status: FMA has faulted this device.
        action: Run 'fmadm faulty' for more information. Clear the errors
                using 'fmadm repaired'.

        c0t5000C5005C04F982d0    UNAVAIL          experienced I/O failures
        status: FMA has degraded this device.
        action: Run 'fmadm faulty' for more information. Clear the errors
                using 'fmadm repaired'.
           see: http://support.oracle.com/msg/ZFS-8000-LR for recovery
$ iostat -en
  ---- errors ---
  s/w h/w trn tot device
    0   0   0   0 c8t0d0
    0  11   0  11 c7t0d0
    0   0   0   0 c0t5000C5005E0C5056d0
    0   0   0   0 c0t5000C5005E169C55d0
    0   0   0   0 c0t5000C5005C08BE07d0
    0   0   0   0 c0t5000C5005E21AE92d0
    0   0   0   0 c0t5000C5005C07780Ad0
    0   0   0   0 c0t5000C5005C04F982d0
  • Please post zpool status -v. – Mark Wagner Apr 17 '15 at 17:39
  • And diskinfo and/or iostat -en. – bahamat Apr 17 '15 at 17:47
  • Amended as promised – Animosity Apr 17 '15 at 22:36
  • What is the CPU doing? 100% on a single core? – Norbert van Nobelen Apr 18 '15 at 18:06
  • Have you fmadm repaired each of the disks, or just the controller? – bahamat Apr 19 '15 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.