I've got a Solaris 11 ZFS-based NAS device with 12x1TB 7.2k rpm SATA drives in a mirrored configuration.
It provides 2 services from the same pool - an NFS server for a small VM farm and a CIFS server for a small team to host shared files. The cifs ZFS has dedup on, while the NFS ZFS filesystem has dedup off. Compression is off everywhere. I'm snapshotting each filesystem every day and keeping the last 14 snapshots.
I've run into a performance issue in cases where I'm either moving, copying or deleting a large amount of data while directly SSH'd into the NAS. Basically, the process seems to block all other IO operations, even to the point of VMs stalling because they receive disk timeouts.
I've a couple of theories as to why this should be the case, but would appreciate some insight into what I might do next.
1) the hardware isn't good enough. I'm not so convinced of this - the system is an HP X1600 (single Xeon CPU) with 30GB RAM. Although the drives are only 7.2k SATA, they should push a max of 80 IOPS each, which should give me more than enough. Happy to be proven wrong though.
2) I've configured it wrong - more than likely. Is it worth turning dedup off everywhere? I'm working under the assumption that RAM = good for dedup, hence giving it a reasonable splodge of RAM.
3) Solaris being stupid about scheduling IO. Is it possible that a local
rm command completely blocks IO to the nfsd? If so, how do I change this?