0

I was running FreeBSD 10.0-RELEASE-p4 and migrated to NAS4Free which is FreeBSD version 10.2-RELEASE #0 r286738M: Thu Aug 13 21:32:58 CEST 2015. Now I'm missing free space and I can't delete any files from the pool. Reading files works fine.

All drives are 2 TB. I didn't remember to export the pool before I migrated. According to internet sources exporting and importing pool should free the space but it doesn't. Pool is running version 15. Could upgrade or scrub help?

NAME        STATE     READ WRITE CKSUM
storage     ONLINE       0     0     0
  raidz2-0  ONLINE       0     0     0
    da0     ONLINE       0     0     0  block size: 512B configured, 4096B native
    da3     ONLINE       0     0     0  block size: 512B configured, 4096B native
    da6     ONLINE       0     0     0  block size: 512B configured, 4096B native
    da9     ONLINE       0     0     0  block size: 512B configured, 4096B native
    da12    ONLINE       0     0     0  block size: 512B configured, 4096B native
    da15    ONLINE       0     0     0  block size: 512B configured, 4096B native
  raidz2-1  ONLINE       0     0     0
    da1     ONLINE       0     0     0  block size: 512B configured, 4096B native
    da4     ONLINE       0     0     0  block size: 512B configured, 4096B native
    da7     ONLINE       0     0     0  block size: 512B configured, 4096B native
    da10    ONLINE       0     0     0  block size: 512B configured, 4096B native
    da13    ONLINE       0     0     0  block size: 512B configured, 4096B native
    da16    ONLINE       0     0     0  block size: 512B configured, 4096B native
  raidz2-2  ONLINE       0     0     0
    da2     ONLINE       0     0     0  block size: 512B configured, 4096B native
    da5     ONLINE       0     0     0  block size: 512B configured, 4096B native
    da8     ONLINE       0     0     0  block size: 512B configured, 4096B native
    da11    ONLINE       0     0     0  block size: 512B configured, 4096B native
    da14    ONLINE       0     0     0  block size: 512B configured, 4096B native
    da17    ONLINE       0     0     0  block size: 512B configured, 4096B native

So: ((6 drives - 2 drives for parity) x 2 TB) x 3 raidz2 = 24 TB which is 21.83 tebibytes usable space minus metadata.

Properties:

# zpool get all
NAME     PROPERTY                       VALUE                          SOURCE
storage  size                           32.6T                          -
storage  capacity                       97%                            -
storage  altroot                        -                              default
storage  health                         ONLINE                         -
storage  guid                           14393350251601347202           local
storage  version                        15                             local
storage  bootfs                         -                              default
storage  delegation                     on                             default
storage  autoreplace                    off                            default
storage  cachefile                      -                              default
storage  failmode                       wait                           default
storage  listsnapshots                  off                            default
storage  autoexpand                     off                            default
storage  dedupditto                     0                              default
storage  dedupratio                     1.00x                          -
storage  free                           696G                           -
storage  allocated                      31.9T                          -
storage  readonly                       off                            -
storage  comment                        -                              default
storage  expandsize                     -                              -
storage  freeing                        0                              local
storage  fragmentation                  -                              -
storage  leaked                         0                              local
storage  feature@async_destroy          disabled                       local
storage  feature@empty_bpobj            disabled                       local
storage  feature@lz4_compress           disabled                       local
storage  feature@multi_vdev_crash_dump  disabled                       local
storage  feature@spacemap_histogram     disabled                       local
storage  feature@enabled_txg            disabled                       local
storage  feature@hole_birth             disabled                       local
storage  feature@extensible_dataset     disabled                       local
storage  feature@embedded_data          disabled                       local
storage  feature@bookmarks              disabled                       local
storage  feature@filesystem_limits      disabled                       local
storage  feature@large_blocks           disabled                       local

zpool iostat shows free space:

# zpool iostat -v
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
storage     31.9T   696G      0      0    550    540
  raidz2    10.8T  63.1G      0      0    162      2
    da0         -      -      0      0    185    135
    da3         -      -      0      0    157    135
    da6         -      -      0      0    158    135
    da9         -      -      0      0    163    135
    da12        -      -      0      0    158    135
    da15        -      -      0      0    149    135
  raidz2    10.8T  90.1G      0      0    162      2
    da1         -      -      0      0    125    101
    da4         -      -      0      0    102    101
    da7         -      -      0      0    115    101
    da10        -      -      0      0    125    101
    da13        -      -      0      0    100    101
    da16        -      -      0      0    106    101
  raidz2    10.3T   543G      0      0    225    536
    da2         -      -      0      0    139    243
    da5         -      -      0      0    133    244
    da8         -      -      0      0    118    246
    da11        -      -      0      0    123    246
    da14        -      -      0      0    123    244
    da17        -      -      0      0    124    243
----------  -----  -----  -----  -----  -----  -----

zpool list shows free space:

# zpool list storage
NAME      SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
storage  32.6T  31.9T   696G         -      -    97%  1.00x  ONLINE  -

zfs list doesn't show free space:

# zfs list
NAME      USED  AVAIL  REFER  MOUNTPOINT
storage  21.3T      0  21.3T  /mnt/storage

df doesn't show free space:

# df -ha
Filesystem      Size    Used   Avail Capacity  Mounted on
/dev/xmd0       120M     69M     51M    58%    /
devfs           1.0K    1.0K      0B   100%    /dev
/dev/xmd1       719M    260M    459M    36%    /usr/local
procfs          4.0K    4.0K      0B   100%    /proc
storage          21T     21T      0B   100%    /mnt/storage
/dev/xmd2       248M    3.9M    224M     2%    /var
tmpfs           128M     48K    128M     0%    /var/tmp
/dev/ada0s1a    743M    151M    592M    20%    /cf

There's no snapshots

# zfs list -t snapshot
no datasets available

Datasets:

# zdb -d storage
Dataset mos [META], ID 0, cr_txg 4, 50.0M, 565 objects
Dataset storage [ZPL], ID 16, cr_txg 1, 21.3T, 1111968 objects
Verified large_blocks feature refcount is correct (0)

Deduplication:

# zdb -D storage
All DDTs are empty

camcontrol:

# camcontrol devlist -v
scbus0 on mpt0 bus 0:
<ATA SAMSUNG HD204UI 0001>         at scbus0 target 0 lun 0 (pass0,da0)
<ATA SAMSUNG HD204UI 0001>         at scbus0 target 1 lun 0 (pass1,da1)
<ATA ST2000DM001-1CH1 CC43>        at scbus0 target 2 lun 0 (pass2,da2)
<ATA SAMSUNG HD204UI 0001>         at scbus0 target 3 lun 0 (pass3,da3)
<ATA SAMSUNG HD204UI 0001>         at scbus0 target 4 lun 0 (pass4,da4)
<ATA ST2000DM001-1CH1 CC43>        at scbus0 target 5 lun 0 (pass5,da5)
<ATA SAMSUNG HD204UI 0001>         at scbus0 target 6 lun 0 (pass6,da6)
<ATA SAMSUNG HD204UI 0001>         at scbus0 target 7 lun 0 (pass7,da7)
<ATA ST2000DM001-1CH1 CC43>        at scbus0 target 8 lun 0 (pass8,da8)
<ATA SAMSUNG HD204UI 0001>         at scbus0 target 9 lun 0 (pass9,da9)
<ATA SAMSUNG HD204UI 0001>         at scbus0 target 10 lun 0 (pass10,da10)
<ATA ST2000DM001-1CH1 CC43>        at scbus0 target 11 lun 0 (pass11,da11)
<ATA SAMSUNG HD204UI 0001>         at scbus0 target 12 lun 0 (pass12,da12)
<ATA SAMSUNG HD204UI 0001>         at scbus0 target 13 lun 0 (pass13,da13)
<ATA ST2000DM001-1CH1 CC43>        at scbus0 target 14 lun 0 (pass14,da14)
<ATA SAMSUNG HD204UI 0001>         at scbus0 target 15 lun 0 (pass15,da15)
<ATA SAMSUNG HD204UI 0001>         at scbus0 target 16 lun 0 (pass16,da16)
<ATA ST2000DM001-1CH1 CC43>        at scbus0 target 17 lun 0 (pass17,da17)
<HP HP SAS EXP Card 2.06>          at scbus0 target 18 lun 0 (pass18,ses0)
<>                                 at scbus0 target -1 lun ffffffff ()
scbus1 on ata2 bus 0:
<>                                 at scbus1 target -1 lun ffffffff ()
scbus2 on ata3 bus 0:
<>                                 at scbus2 target -1 lun ffffffff ()
scbus3 on ata4 bus 0:
<KINGSTON SV300S37A60G 525ABBF0>   at scbus3 target 0 lun 0 (ada0,pass19)
<>                                 at scbus3 target -1 lun ffffffff ()
scbus4 on ata5 bus 0:
<>                                 at scbus4 target -1 lun ffffffff ()
scbus-1 on xpt0 bus 0:
<>                                 at scbus-1 target -1 lun ffffffff (xpt0)
0

FreeBSD 10.2 added tunable, vfs.zfs.spa_slop_shift, which controls how much space is reserved by default.

Source: https://www.freebsd.org/releases/10.2R/relnotes.html#storage-zfs

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.