1

I just expanded my zpool named storage with 3 HGST 6TB drives in raidz1. I thought this would add only about 11.x TB of storage (since they are really 6000GB drives) how ever now zpool list shows I have 27.1 TB total. This seems more than I should have. I am terrified I did some thing wrong and I have lost parity on my home server array. I added them to the pool with

sudo zpool add -f storage raidz1 drive1 drive2 drive3

Have I done something wrong and if so is it possible to revert my change?

sudo zpool list

NAME      SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
storage  27.1T  9.32T  17.8T         -      -    34%  1.00x  ONLINE  -

sudo zpool status

NAME                                          STATE     READ WRITE CKSUM
    storage                                       ONLINE       0     0     0
      mirror-0                                    ONLINE       0     0     0
        ata-WDC_WD60EFRX-68MYMN1_WD-WX11DA40H4ZK  ONLINE       0     0     0
        ata-WDC_WD60EFRX-68MYMN1_WD-WX51D6410414  ONLINE       0     0     0
      mirror-1                                    ONLINE       0     0     0
        ata-WDC_WD60EFRX-68MYMN1_WD-WX91D65355TK  ONLINE       0     0     0
        ata-WDC_WD60EFRX-68L0BN1_WD-WXB1HB4TP7L0  ONLINE       0     0     0
      raidz1-2                                    ONLINE       0     0     0
        wwn-0x5000cca255c02505                    ONLINE       0     0     0
        wwn-0x5000cca255c039a7                    ONLINE       0     0     0
        wwn-0x5000cca255c03b95                    ONLINE       0     0     0

errors: No known data errors

sudo fdisk -l

WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sda: 6001.2 GB, 6001175126016 bytes
256 heads, 63 sectors/track, 726751 cylinders, total 11721045168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdc: 250.1 GB, 250059350016 bytes**This is the OS Drive**
255 heads, 63 sectors/track, 30401 cylinders, total 488397168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xcb1189ee

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1   488397167   244198583+  ee  GPT

WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdb: 6001.2 GB, 6001175126016 bytes
256 heads, 63 sectors/track, 726751 cylinders, total 11721045168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdd: 6001.2 GB, 6001175126016 bytes
256 heads, 63 sectors/track, 726751 cylinders, total 11721045168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

WARNING: GPT (GUID Partition Table) detected on '/dev/sde'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sde: 6001.2 GB, 6001175126016 bytes
256 heads, 63 sectors/track, 726751 cylinders, total 11721045168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

WARNING: GPT (GUID Partition Table) detected on '/dev/sdf'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdf: 6001.2 GB, 6001175126016 bytes
256 heads, 63 sectors/track, 726751 cylinders, total 11721045168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdf1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

WARNING: GPT (GUID Partition Table) detected on '/dev/sdg'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdg: 6001.2 GB, 6001175126016 bytes
256 heads, 63 sectors/track, 726751 cylinders, total 11721045168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdg1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

WARNING: GPT (GUID Partition Table) detected on '/dev/sdh'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdh: 6001.2 GB, 6001175126016 bytes
256 heads, 63 sectors/track, 726751 cylinders, total 11721045168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdh1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

sudo lshw -class disk

        *-disk:0
         description: ATA Disk
         product: HGST HDN726060AL
         physical id: 0.0.0
         bus info: scsi@0:0.0.0
         logical name: /dev/sdf
         version: T517
         serial: K1G0HW1B
         size: 5589GiB (6001GB)
         capacity: 5589GiB (6001GB)
         capabilities: 15000rpm gpt-1.00 partitioned partitioned:gpt
         configuration: ansiversion=6 guid=62a489e5-51af-c94f-9535-235b7bbe2d15 sectorsize=4096
    *-disk:1
         description: ATA Disk
         product: HGST HDN726060AL
         physical id: 0.1.0
         bus info: scsi@0:0.1.0
         logical name: /dev/sdg
         version: T517
         serial: K1G09VRB
         size: 5589GiB (6001GB)
         capacity: 5589GiB (6001GB)
         capabilities: 15000rpm gpt-1.00 partitioned partitioned:gpt
         configuration: ansiversion=6 guid=a7295fde-f0f8-7f41-8546-7ee02490d72b sectorsize=4096
    *-disk:2
         description: ATA Disk
         product: HGST HDN726060AL
         physical id: 0.2.0
         bus info: scsi@0:0.2.0
         logical name: /dev/sdh
         version: T517
         serial: K1G0HB3B
         size: 5589GiB (6001GB)
         capacity: 5589GiB (6001GB)
         capabilities: 15000rpm gpt-1.00 partitioned partitioned:gpt
         configuration: ansiversion=6 guid=4677c5c9-364a-c943-a8d5-90d2159f3cf0 sectorsize=4096
    *-disk
         description: ATA Disk
         product: WDC WD60EFRX-68M
         vendor: Western Digital
         physical id: 0.0.0
         bus info: scsi@1:0.0.0
         logical name: /dev/sda
         version: 82.0
         serial: WD-WX91D65355TK
         size: 5589GiB (6001GB)
         capabilities: gpt-1.00 partitioned partitioned:gpt
         configuration: ansiversion=5 guid=e4026c6a-61c5-9a4a-adaf-ebcdf8bba68c sectorsize=4096
    *-disk
         description: ATA Disk
         product: WDC WD60EFRX-68L
         vendor: Western Digital
         physical id: 0.0.0
         bus info: scsi@2:0.0.0
         logical name: /dev/sdb
         version: 82.0
         serial: WD-WXB1HB4TP7L0
         size: 5589GiB (6001GB)
         capabilities: gpt-1.00 partitioned partitioned:gpt
         configuration: ansiversion=5 guid=8b576966-3d94-424b-aae9-9fb35c0e070a sectorsize=4096
    *-disk
         description: ATA Disk
         product: Samsung SSD 840
         physical id: 0.0.0
         bus info: scsi@4:0.0.0
         logical name: /dev/sdc
         version: EXT0
         serial: S1DBNEADA00610E
         size: 232GiB (250GB)
         capabilities: gpt-1.00 partitioned partitioned:gpt
         configuration: ansiversion=5 guid=8d48ca17-2e53-4ee4-a040-e80362e68eca sectorsize=512
    *-disk
         description: ATA Disk
         product: WDC WD60EFRX-68M
         vendor: Western Digital
         physical id: 0.0.0
         bus info: scsi@5:0.0.0
         logical name: /dev/sdd
         version: 82.0
         serial: WD-WX11DA40H4ZK
         size: 5589GiB (6001GB)
         capabilities: gpt-1.00 partitioned partitioned:gpt
         configuration: ansiversion=5 guid=5811c203-99cd-5342-bb8f-c838f896b64e sectorsize=4096
  *-disk
         description: ATA Disk
         product: WDC WD60EFRX-68M
         vendor: Western Digital
         physical id: 0.0.0
         bus info: scsi@6:0.0.0
         logical name: /dev/sde
         version: 82.0
         serial: WD-WX51D6410414
         size: 5589GiB (6001GB)
         capabilities: gpt-1.00 partitioned partitioned:gpt
         configuration: ansiversion=5 guid=c24e5950-cd95-f94b-826a-21a004221d32 sectorsize=4096
3

Oh no! You did something kinda bad...

Please read: http://nex7.blogspot.com/2013/03/readme1st.html

Two issues to note:

Under RAIDZ1|2|3, the zpool command reflects the size of the disks comprising the pool, including parity. zfs list will show usable space. That's where the discrepancy comes from.

Also, what you ended up doing was... expanding a 4-disk RAID 1+0 (ZFS mirrors) by adding a RAIDZ1 group of 3 disks.

So your mirrors mean that the RAID1 pairs are striped together. Fine. But by adding a RAIDZ1 group, you now have a stripe across two mirror sets and a RAIDZ1.

Failure of either two disks in a mirror group or two disks in the RAIDZ1 group would result in total pool failure.

You should have added an even number if disks by creating more mirror pairs. Right now, you have no option to revert the change because your data is scattered across the disk groups. This is probably a backup.rebuild.restore situation.

10
  • First of all thank you! Second could you explain in more detail which drives would need to fail for total pool loss. What you described seems fine. should I only ever add more groups of the same number of disks? – NathanTheGr8 May 22 '16 at 5:33
  • You can lose one disk per VDEV and still run. If any two disks in an particular VDEV, fail, the pool fails. You really need to rebuild this. – ewwhite May 22 '16 at 8:45
  • That sounds what I intended. Why is that so bad? Each vdev should only have one disk of parity fault. – NathanTheGr8 May 22 '16 at 21:10
  • Your VDEVs need to be identical and of the same type. Please read: Things Nobody Told You About ZFS. – ewwhite May 22 '16 at 21:12
  • thank you for the blog post, I did see the part where he recommended identical vdevs but he didn't mention why. He talked about performance being better for 2 disks mirrors, but I didn't see any explanation for the identical vdevs. – NathanTheGr8 May 22 '16 at 22:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.