I have this ZFS pool consisting of two mirrored 1TB Samsung 870 QVO drives, that I've been using for about a year for storing bhyve guest images as ZVOLs. During that time, I've created / messed with / destroyed numerous VM instances; this pool has seen quite a lot of disk activity.
Last night, while looking at disk I/O performance on one of the VMs, I realized that I had never trimmed these SSDs before. Not really knowing what to expect, I shut down the VMs and ran zfs trim on the pool.
Overall, the whole operation took a whopping 16 hours (!) to complete. For reference, here's what gstat -pdo had to say during that time (typical output, with values averaged over 10s):
L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d o/s ms/o %busy Name
11 45 0 0 0.0 27 292 84.9 18 8497 114.3 0 82.9 101.0| ada0
12 37 0 0 0.0 26 283 92.1 11 6547 189.1 0 84.2 100.6| ada1
In any case, I patiently let it run to completion.
Then, earlier today and with all the VMs still stopped, I decided to run zfs trim again. My reasoning was that, as there had been no significant activity on that pool since the last TRIM, and with most of the unallocated blocks supposedly reclaimed by the SSD controller the first time around, it would be much faster this time.
Well, my intuition was wrong: this TRIM has now been running for about 5 hours, and according to zpool status -t, it's just over 30% done. So apparently, I'm looking at a similar overall run time (~16h, give or take).
Is that behavior expected ? Because if it is, then obviously I must be missing something about how TRIM works.
Notes:
- I am aware of the shortcomings of the QVO series (4-bit MLC flash, notoriously small amount of pseudo-SLC cache), so I wasn't expecting any crazy performance to begin with.
- Maybe there's something wrong with my setup (bad SATA cable, etc..) but still, that doesn't explain why the system doesn't seem to benefit from previous TRIM runs on the same pool.
