I have encountered to be what I consider a very strange performance issue
I have an application that processes some data and creates 100GB or so of files. This runs fine, and I can scp the files off the machine afterwards. Local disks are ZFS, Maybe that's the problem.
If however I run another utility that mmaps the files and swaps the bytes of the files around, before copying the files off, the files: 1) transfer much slower, maybe 50mb/s (or worse) off a solid state disk, systat -iostat shows a much higher disk transactions level. 2) the Wired memory of the machine goes through the roof, to accommodate a 70+ GB ARC A 70GB arc seems ridiculous since that's around a quarter of the local disk storage. If after transfer the files are deleted, the ARC drops, but the wired memory doesn't seem to. Any ideas on what is going on?