We have about 10TB of files that we use on our fileserver. I'll be setting up a new server with more space and better backups. We have a SAS box with 24 drives connected to OpenIndiana server configured as a pool of 11 mirrored drives with two spares. Is it better to present the pool of drives as a iscsi target to a Windows fileserver directly and store files that way or connect it to a exsi server and store the data within vmdk disks?
You can do both.
A major point of using ZFS is to give you maximum storage flexibility.
For example, I have the following set of ZFS filesystems, where the first four listed are NFS mounts exported to a VMware vSphere 5.1 cluster... but the last export is an iSCSI export mounted directly by a virtual machine. It's all backed by the same storage, but you can do whatever is needed by your application or environment.
In practice, I tend to use ZFS NFS filesystems for VMware and iSCSI for backup storage or where I need a particular native filesystem presented to an OS (e.g. NTFS).
If you give your file server its storage through VMFS, you will have more flexibility with VMWare. They allow hot VMotion with raw devices, however storage VMotion won't work.