Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Today I have found this on a server (FreeBSD 8.2 STABLE):

data DEGRADED 1.38K 0 0
  raidz1-0 DEGRADED 1.38K 0 0
    ad10 ONLINE 1.38K 0 0
    ad12 ONLINE 0 0 0
    ad14 ONLINE 0 0 0
    ad16 REMOVED 0 0 0 

I quickly pulled down the wrong HDD and put in a new one. After that, I typed in this unlucky command:

zpool add data ad16

The result was that a new ad16 device appeared in the pool:

data        DEGRADED     0     0     0
  raidz1-0  DEGRADED     0     0     0
    ad10    ONLINE       0     0     0
    ad12    ONLINE       0     0     0
    ad14    ONLINE       0     0     0
    ad16    FAULTED      0     0     0  corrupted data
  ad16      ONLINE       0     0     0

The first ad16 device is FAULTED and part of data/raidz1-0 volume. The second ad16 device is ONLINE, and not part of any volume. The problem is that they have the same name so the replace command does not work:

gw# zpool replace -f data ad16 ad16
invalid vdev specification
the following errors must be manually repaired:
/dev/ad16 is part of active pool 'data'

I think that I should remove the ONLINE ad16 disk before I could replace the FAULTED ad16 disk. But this doesn't work because I'm not able to put it offline, nor remove it:

gw# zpool offline data ad16
gw# zpool status
  pool: data
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
action: Online the device using 'zpool online' or replace the device with
    'zpool replace'.
 scan: scrub in progress since Thu Apr 18 03:23:06 2013
    26.1G scanned out of 3.13T at 50.7M/s, 17h52m to go
    0 repaired, 0.81% done

    data        DEGRADED     0     0     0
      raidz1-0  DEGRADED     0     0     0
        ad10    ONLINE       0     0     0
        ad12    ONLINE       0     0     0
        ad14    ONLINE       0     0     0
        ad16    OFFLINE      0     0     0
      ad16      ONLINE       0     0     0

errors: No known data errors
gw# zpool remove data ad16
cannot remove ad16: only inactive hot spares, cache, top-level, or log devices can be removed

I guess that the "offline ad16" command is targeting the FAULTED device. But I would like to offline the other one. I have also tried to boot the system into single user mode, with this new disk removed, but it resulted in both ad16 devices being UNAVAIL, and the whole pool unusable (which is strange, because there are enough disks to make it work...)

share|improve this question
Also, I don't understand the logic here. Why offline ad16 is targeting the device that is already OFFLINE? Or, if these commands are targeting the first device with that name, then why remove ad16 is not doing that? – nagylzs Apr 19 '13 at 6:49
up vote 2 down vote accepted

The, 'zpool add' command is for adding new devices (vdevs) to pools. When you originally ran that command, you added a new vdev (consisting of only ad16) to the pool. You started with a 4 disk raidz, with one failed disk, but now you have a pool where data is striped between the raidz and ad16. Losing that ONLINE ad16 disk will FAULT the entire pool.

As it's not possible to remove ad16 now (you cannot remove a vdev from a pool), and I doubt you want your data striped between a 4-disk raidz and a single disk, I don't think it's worth your time trying to sort out that FAULTED disk. You'd be better off looking at getting your data off onto a separate pool/disk/server and destroy/recreate this pool.

share|improve this answer
That was a terrible mistake, I see now. The problem was in my mind. I didn't know that data is stripped between "raidz1-0" and ad16. Thanks. – nagylzs Apr 19 '13 at 7:41

Suggestion: Reference the underlying device by the UUID vs the logical name.


share|improve this answer
How do I do that for a disk that is not physically in the system? I already have gptid turned on: gw# sysctl kern.geom.label.gptid.enable kern.geom.label.gptid.enable: 1 But the devices do not appear with their gptid. – nagylzs Apr 19 '13 at 7:35
Ouch. :( @USD Matt's answer is definitely right on: migrate the data off/rebuild. – Wing Tang Wong Apr 19 '13 at 15:59

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.