0

I recently expanded my storage pool "zstorage" by adding two more drives. It had 3 3TB drives in RAID-Z, and I added two more 1.5TB drives also in RAID-Z (which I understand is effectively the same as mirroring). I know it's not the most efficient setup, but these are the drives I had. I expect that with parity, I should get to use 6TB from the first set and 1.5TB from the second for 7.5TB total.

After the add, everything looks fine, and from the commands below, you can see that there is now more space listed by zpool list, and the drives show up as I would expect with zpool status. However, the amount of disk space available did not increase, as you can see below.

Any idea why I did not get more disk space out of this? I seem to have gained nothing from adding the two 1.5TB drives.

kevin@atlas:~$ sudo zpool list

    NAME       SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
    zstorage  10.8T  7.79T  3.05T    71%  1.00x  ONLINE  -

kevin@atlas:~$ sudo zfs list -t all

    NAME       USED  AVAIL  REFER  MOUNTPOINT
    zstorage  5.19T  1.48T  5.19T  /zstorage

kevin@atlas:~$ sudo zpool status -v

      pool: zstorage
     state: ONLINE
      scan: scrub in progress since Sat Apr  5 10:49:13 2014
        317G scanned out of 7.79T at 262M/s, 8h18m to go
        1.50M repaired, 3.97% done
    config:

            NAME                                          STATE     READ WRITE CKSUM
            zstorage                                      ONLINE       0     0     0
              raidz1-0                                    ONLINE       0     0     0
                ata-WDC_WD30EZRX-00DC0B0_WD-WCC1T1735698  ONLINE       0     0     0  (repairing)
                ata-WDC_WD30EZRX-00DC0B0_WD-WMC1T0506289  ONLINE       0     0     0
                ata-WDC_WD30EZRX-00MMMB0_WD-WCAWZ2711600  ONLINE       0     0     0
              raidz1-1                                    ONLINE       0     0     0
                ata-WDC_WD15EADS-00P8B0_WD-WMAVU0454800   ONLINE       0     0     0
                ata-WDC_WD15EADS-00P8B0_WD-WMAVU0524642   ONLINE       0     0     0

    errors: No known data errors

kevin@atlas:~$ df -h /zstorage/

    Filesystem      Size  Used Avail Use% Mounted on
    zstorage        6.7T  5.2T  1.5T  78% /zstorage
Improve this question
6
  • are you using zfs quota?
    – c4f4t0r
    Apr 5, 2014 at 20:40
  • Wow... Take a good read of nex7.blogspot.com/2013/03/readme1st.html
    – ewwhite
    Apr 6, 2014 at 22:23
  • Why is there a RAIDZ1 comprised of two disks?
    – ewwhite
    Apr 6, 2014 at 22:25
  • 1
    Regardless of the reason, what you've done here was stupid and ill-advised. Use equal size disks! You can't just add random drives to the array like this. A two-disk raidz1 makes no sense. You've effectively reduced the performance and resiliency of your dataset by striping across a trio of disks and a pair of drives. This is completely a wipe-and-rebuild situation and it can't be undone.
    – ewwhite
    Apr 7, 2014 at 2:15
  • I have 3 3TB disks and 2 1.5TB disks. This is for home use, and I don't intend to purchase any more disks. Now, that said, I absolutely would like to have the best configuration. :) Two-disk RAIDZ-1 is effectively the same as mirroring, is it not? I chose raidz simply because the other vdev was raidz, and I thought it didn't matter. Would 5-disk raidz-1 be the best option? I assume so, but this wasn't an option for me now since I started with 3-disk raidz. The problem seems out to be autoexpand, by the way. It was turned off, and I wasn't aware of the option.
    – Kevin Wood
    Apr 7, 2014 at 11:03

2 Answers 2

Reset to default
5

Everything is operating normally. You're running into TB vs TiB confusion.

TL;DR

  • 12TB raw == 10.8TiB raw (which is what is listed by zpool list)
  • 7.5TB usable == 6.8TiB usable (which is what is listed by zfs list)

You have:

  • 3×3TB drives
  • 2×1.5TB drives

yielding a total of 12TB of raw storage.

○ → units 12TB TiB
    * 10.913936
    / 0.091625969

Which is roughly the space you get in zpool list:

NAME       SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
zstorage  10.8T  7.79T  3.05T    71%  1.00x  ONLINE  -

As for usable:

  • 6TB (raidz1-0)
  • 1.5TB (raidz1-1)

In TiB:

○ → units 7.5TB TiB
    * 6.8212103
    / 0.14660155

You have:

NAME       USED  AVAIL  REFER  MOUNTPOINT
zstorage  5.19T  1.48T  5.19T  /zstorage

The zstorage filesystem has 5.19TiB used and 1.48TiB available (6.67TiB total). Close enough to 6.82TiB for me (there will be some overhead).

Improve this answer
4
  • Nope. It's not a TB versus TiB issue. zpool list will show the raw amount of pool space on raidz(2) volumes.
    – ewwhite
    Apr 7, 2014 at 2:10
  • Please clarify. TB vs TiB confusion explains the situation 100%.
    – MikeyB
    Apr 7, 2014 at 2:23
  • As I understand, the commands show the raw free space, not counting for anything that will be used for parity. Writing 2MB would actually consume 3MB with parity. In any case, I can say with certainty that the "free" space did not increase when I added the new disks, but the "available" space did.
    – Kevin Wood
    Apr 7, 2014 at 11:08
  • Oh man, I did have it all wrong. The space is there. I was just confused. Sorry all...
    – Kevin Wood
    Apr 7, 2014 at 14:59
3

You should check that the auto expand property is set for your zpool.

$ zpool get autoexpand zstorage

If the property is set to off, you should set it to on to have your pool automatically expand to fill the available space.

$ zpool set autoexpand=off zstorage

I'll include the relevant portion of the man page for clarity.

 autoexpand=on | off

     Controls automatic pool expansion  when  the  underlying
     LUN  is  grown.  If  set to on, the pool will be resized
     according to the size of the  expanded  device.  If  the
     device  is  part  of  a mirror or raidz then all devices
     within that mirror/raidz group must be  expanded  before
     the new space is made available to the pool. The default
     behavior is off. This property can also be  referred  to
     by its shortened column name, expand.

I believe you can also use 

$ zpool online -e zstorage

to instruct the pool to manually expand, but I have not used that command myself. I'm not sure if the pool would need to be offlined first (I would expect it to be unnecessary).

 zpool online [-e] pool device...

     Brings the specified physical device online.

     This command is not applicable to spares or  cache  dev-
     ices.

     -e

         Expand the device to use all available space. If the
         device is part of a mirror or raidz then all devices
         must be expanded before the new  space  will  become
         available to the pool.
Improve this answer
3
  • Incorrect - the autoexpand feature handles the case when the underlying LUN expands, not when you add new LUNs.
    – MikeyB
    Apr 7, 2014 at 1:57
  • Well, autoexpand was off, which was the default configuration. I turned it on, ran zpool online -e zstorage <disk> for every disk, and ran zpool export and zpool import. After all of this, the problem still remains. It won't let me zpool online the vdevs "raidz1-0" and "raidz1-1", so I can't expand these. So in the end, I'm not sure that this is the solution.
    – Kevin Wood
    Apr 7, 2014 at 11:14
  • Right, since your LUN sizes didn't change. If you shut down your system, replaced your 3TB disks with 4TB disks, dded the data over to the 4TB disks, then turned everything back on this would be applicable. Or, more conventionally, expanded your LV or the LUN provided by your storage subsystem. :)
    – MikeyB
    Apr 7, 2014 at 18:14

Not the answer you're looking for? Browse other questions tagged .