Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I use ZFS in FreeNAS 8.0.4 and use iSCSI to provide a LUN to Vmware ESXi 5.0.

I post it at http://forums.freenas.org/showthread.php?10990-Memory-utilization-and-performance-problem/page4, but I believe people here may have more expertise on ZFS.

FreeNAS doesn't have iSCSI unmap. I have create / move / delete virtual machine between ZFS iSCSI. After a while, zpool list report 95% capacity used. However, Vmware shows VMFS only uses 50% capacity.

AFAIK, when creating a virtual machine, zvol allocated disk space. However, when we moved / deleted virtual machine, zvol didn't update / didn't aware to update the free capacity.

[root@data1] ~# zpool list
data1 2.25T 2.07T 185G 95% ONLINE /mnt

Now, I believe due to the wrong free capacity reported in zpool list, the performance is deeply affected.

How I should do to let ZFS know the real free capacity.

Thanks million.

zpool status -v

  pool: data1
 state: ONLINE
 scrub: scrub stopped after 10h31m with 0 errors on Wed Feb  6 13:32:53 2013

        NAME        STATE     READ WRITE CKSUM
        data1       ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ada0p2  ONLINE       0     0     0
            ada1p2  ONLINE       0     0     0
            ada2p2  ONLINE       0     0     0
            ada3p2  ONLINE       0     0     0
            ada4p2  ONLINE       0     0     0
          ada5p2    AVAIL   

errors: No known data errors

zpool list

data1  2.25T  2.08T   179G    92%  ONLINE  /mnt
share|improve this question
What's the actual problem you're having?? –  Chris S Feb 18 '13 at 15:46
add comment

2 Answers

You made your zvol too large and your ZFS implementation doesn't have the ability to unmap release blocks!

The best way to prevent this from happening is to underprovision the LUNs you present to VMware. Since we know ZFS performance degrades when the disk is near full, it's best to limit the maximum size of your zvol.

e.g. if you have 200GB zpool storage, make sure the zvols you present to VMware are less than that - maybe 150GB.

ZFS is unaware of the filesystem being using on top of the zvol block device, so all that's happening on your VMFS volume is that it knows that it has 50% free space. ZFS has no way of knowing that. You will not be able to reclaim the space without reformatting/recreating the volume or expanding the zvol.

A better solution in your case would be using NFS instead of iSCSI for this purpose. If you're not monitoring VM disk utilization and growth carefully enough to prevent this from happening with iSCSI, NFS may make more sense. I use NFS with my ZFS+VMware setups.

Also see: When using thin-provisioning with ZFS, how do you make sure you don't run out of physical disk space?


Incorrect Used Space reported by zVol.

share|improve this answer
Thanks for your tip of making zvol smaller than zpool. Now, I only want to reclaim the space to increase performance a bit. Then move to another zfs. It's too slow to move now. –  Datnus Feb 11 '13 at 3:08
I don't think you have any options. –  ewwhite Feb 11 '13 at 3:21
I still have 1 spare disk. If I add it to zpool, will it help? Will the performance be affected to add new HDD to zpool? –  Datnus Feb 11 '13 at 3:27
Depends. Show us the output of zpool status -v. –  ewwhite Feb 11 '13 at 3:34
Bummer. RAIDZ can't be expanded. –  ewwhite Feb 11 '13 at 4:39
show 7 more comments

You have rightly identified the cause - and by the by, even with SCSI UNMAP, it likely would have still occurred, as the support matrix right now for TRIM and UNMAP is abysmal. However, there is a solution nobody has touched on.

First - enable compression (compress=on or compress=lzjb, I'm not a fan of compress=gzip, especially for VM's), none of this will work without that.

If you're dealing with drive(s) offered up into a VM (or as the VM's OS drive, too) and then formatted with a filesystem, you can use utilities such as "zerofree" for Linux or "sdelete" for Windows to write zeroes out to all the free blocks. Due to a quirk in how ZFS handles blocks of 0's on compressed filesystems, this will free up the space on the zvol, getting the zvol much more in line with the actual used capacity.

If you've done the above and it didn't work (unlikely) or there's still a very large discrepancy (also unlikely), even after running this on all drives on all VM's sitting on top of the LUN, you might try this article - note I haven't tested this, at all, so no promises: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2014849

share|improve this answer
FWIW, at least Solaris also supports zfs set compression=zle ... which enables "zero length encoding" by just writing zero block pointers for zero-filled blocks. It does not do any other compression, thus the impact on the CPU is minimal. I do not know if the BSD implementation includes support for zle though. –  the-wabbit Oct 13 '13 at 7:53
add comment

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.