2

I use ZFS in FreeNAS 8.0.4 and use iSCSI to provide a LUN to Vmware ESXi 5.0.

I post it at http://forums.freenas.org/showthread.php?10990-Memory-utilization-and-performance-problem/page4, but I believe people here may have more expertise on ZFS.

FreeNAS doesn't have iSCSI unmap. I have create / move / delete virtual machine between ZFS iSCSI. After a while, zpool list report 95% capacity used. However, Vmware shows VMFS only uses 50% capacity.

AFAIK, when creating a virtual machine, zvol allocated disk space. However, when we moved / deleted virtual machine, zvol didn't update / didn't aware to update the free capacity.

[root@data1] ~# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
data1 2.25T 2.07T 185G 95% ONLINE /mnt

Now, I believe due to the wrong free capacity reported in zpool list, the performance is deeply affected.

How I should do to let ZFS know the real free capacity.

Thanks million.


zpool status -v

  pool: data1
 state: ONLINE
 scrub: scrub stopped after 10h31m with 0 errors on Wed Feb  6 13:32:53 2013
config:

        NAME        STATE     READ WRITE CKSUM
        data1       ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ada0p2  ONLINE       0     0     0
            ada1p2  ONLINE       0     0     0
            ada2p2  ONLINE       0     0     0
            ada3p2  ONLINE       0     0     0
            ada4p2  ONLINE       0     0     0
        spares
          ada5p2    AVAIL   

errors: No known data errors

zpool list

NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
data1  2.25T  2.08T   179G    92%  ONLINE  /mnt
1
  • What's the actual problem you're having??
    – Chris S
    Feb 18, 2013 at 15:46

3 Answers 3

2

You made your zvol too large and your ZFS implementation doesn't have the ability to unmap release blocks!

The best way to prevent this from happening is to underprovision the LUNs you present to VMware. Since we know ZFS performance degrades when the disk is near full, it's best to limit the maximum size of your zvol.

e.g. if you have 200GB zpool storage, make sure the zvols you present to VMware are less than that - maybe 150GB.

ZFS is unaware of the filesystem being using on top of the zvol block device, so all that's happening on your VMFS volume is that it knows that it has 50% free space. ZFS has no way of knowing that. You will not be able to reclaim the space without reformatting/recreating the volume or expanding the zvol.

A better solution in your case would be using NFS instead of iSCSI for this purpose. If you're not monitoring VM disk utilization and growth carefully enough to prevent this from happening with iSCSI, NFS may make more sense. I use NFS with my ZFS+VMware setups.

Also see: When using thin-provisioning with ZFS, how do you make sure you don't run out of physical disk space?

and

Incorrect Used Space reported by zVol.

12
  • Thanks for your tip of making zvol smaller than zpool. Now, I only want to reclaim the space to increase performance a bit. Then move to another zfs. It's too slow to move now.
    – Datnus
    Feb 11, 2013 at 3:08
  • I don't think you have any options.
    – ewwhite
    Feb 11, 2013 at 3:21
  • I still have 1 spare disk. If I add it to zpool, will it help? Will the performance be affected to add new HDD to zpool?
    – Datnus
    Feb 11, 2013 at 3:27
  • Depends. Show us the output of zpool status -v.
    – ewwhite
    Feb 11, 2013 at 3:34
  • Bummer. RAIDZ can't be expanded.
    – ewwhite
    Feb 11, 2013 at 4:39
2

You have rightly identified the cause - and by the by, even with SCSI UNMAP, it likely would have still occurred, as the support matrix right now for TRIM and UNMAP is abysmal. However, there is a solution nobody has touched on.

First - enable compression (compress=on or compress=lzjb, I'm not a fan of compress=gzip, especially for VM's), none of this will work without that.

If you're dealing with drive(s) offered up into a VM (or as the VM's OS drive, too) and then formatted with a filesystem, you can use utilities such as "zerofree" for Linux or "sdelete" for Windows to write zeroes out to all the free blocks. Due to a quirk in how ZFS handles blocks of 0's on compressed filesystems, this will free up the space on the zvol, getting the zvol much more in line with the actual used capacity.

If you've done the above and it didn't work (unlikely) or there's still a very large discrepancy (also unlikely), even after running this on all drives on all VM's sitting on top of the LUN, you might try this article - note I haven't tested this, at all, so no promises: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2014849

1
  • FWIW, at least Solaris also supports zfs set compression=zle ... which enables "zero length encoding" by just writing zero block pointers for zero-filled blocks. It does not do any other compression, thus the impact on the CPU is minimal. I do not know if the BSD implementation includes support for zle though.
    – the-wabbit
    Oct 13, 2013 at 7:53
0

As you said that

FreeNAS doesn't have iSCSI unmap

maybe you could look into TRIM/UNMAP Zvol over iSCSI in which I asked for help to do exact this: "Reclaim free space on a zfs emulated volume."

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .