ZFS isn't a cluster file system, that's for sure, so can be used for single-server access but not for multiple. That said it makes a wonderful basis for a non-clustered NAS.
As for the Nexenta question - yes you can scale it very well from small to very large quite easily.
For performance I'd be tempted to stick to a single 4-6 core CPU, you won't benefit from many more cores. Try to get at least 4GB so you'll have a nice large cache. Use Dual teamed NICs. When picking controllers pick a name brand, you can't really go wrong with Adaptec, make sure it supports exposing all disks raw when in JBOD mode (i.e. if you have 16 drives does it appear as 16 drives, what's the most of these supported per controller). Always go SAS rather than SATA (although those new RE3 disks are quite nice and are genuine 24/365 drives too). Make sure you get dual power supplies. If possible use dual-port SAS disks and ensure both paths are live. Setup whatever alerting/monitoring you can with the controller/s to let you know IMMEDIATELY when you have a dead disk etc. Oh and buy at least one spare disk to be kept 'cold' and if you can have another set as 'hot' standby.
Other than that get on with it :)