I have read that ZFS stripes the data in a zpool across all top-level vdevs, assuming all vdevs are added at the beginning of the pool's life. Everything I have read seems to regard this as a good thing. But, it seems to me like for deployments with many disks, this is not going to lead to good overall performance from all those disks in a multi-user (or even just multi-process) environment.
Suppose for example I have 96 disks, which I use to create 12 vdevs of 8 disks each, all of which I add to my zpool. Then I set it loose on users and they fill it up with all manner of craziness. Some files are tens of gigabytes, others are small user application configuration files etc.
Later, user A wants to copy some multi-gigabyte files. She starts an rsync or somesuch, and experiences blazing performance from the underlying sequential reads off the 12 striped vdevs. But then user B fires up another application that also requests fairly large chunks of data at a time. Now drive heads are constantly getting pulled off user A's rsync to deal with user B, and although each application is individually relatively sequential, the 96 disks are all involved in both users' requests, and see seek patterns and performance more consistent with random I/O.
In this 12 vdevs of 8 disks configuration, each vdev still has 8 disks' worth of performance, so I'd expect sequential I/O to be very good even without additional striping across other vdevs. Wouldn't it be better for ZFS to put many gigabytes on one vdev before moving on to another one? (In my experiments I'm getting stripes around 500k.) In that way, user A's reads would only have a 1/12 chance of using the same disks as user B's reads, and they'd both get performance consistent with sequential I/O most of the time.
Is there a way to get good performance from ZFS in this configuration/workload?