Context: I'm on a Toshiba 512 GB NVMe (Model: KXG50ZNV512G)

I'm seeing this weird behaviour while benchmarking Postgres on ZFS-on-Linux (via pgbench), where the second and third runs of a benchmark are progressively slower than the first run.

Here is what is happening:

client=1  |  770 =>  697 | 10% reduction in TPS
client=4  | 2717 => 2180 | 24% reduction in TPS
client=8  | 4579 => 3339 | 37% reduction in TPS
client=12 | 4219 => 4175 | 01% reduction in TPS
client=48 | 5902 => 5623 | 05% reduction in TPS
client=96 | 7094 => 6739 | 05% reduction in TPS

I'm re-running these tests and the early numbers indicate that the 3rd run is slower than the 1st and 4th is slower than the 3rd.

Could the lack of TRIM support on ZFS-on-Linux causing this - https://github.com/zfsonlinux/zfs/pull/8255 ?

  • Doubt that, that would be a speed of light degrading. This happens in a more slow way. – drookie Jan 30 at 17:08

Rather than the missing TRIM support (whose performance deficit you can often avoid by simply leaving ~10% unpartitioned space at the end of the disk), what is hitting you probably is ZFS CoW behavior.

Basically, when running on an empty dataset, you can write without incurring in read/modify/write because, well, you have not written much yet. When really rewriting data (as in following benchmarks), you are going to progressively hit more and more read/modify/write, leading to both read and write amplification (and slower performance).

To check if it is the case, simply use zpool iostat to record total reads/writes on the first three runs: if you see the second and third to command an increased amount of transferred bytes, you have the confirmation of what written above.

  • Do you confuse CoW behavior for 512/4k sectors unalignment ? Because it looks that way. In the sane time sector unalignment cannot cause progressive degrading. – drookie Jan 30 at 17:01
  • @drookie no, sector misalignment is avoided with proper ashift settings. I'm referring to r/m/w amplification which occurs when using recordsize larger than actual write i/o – shodanshok Jan 30 at 17:28
  • Well, this is simply avoided by setting the recordsize matching the db block size. But I doubt this can cause the progressing degradation too. – drookie Jan 30 at 19:18
  • @drookie it is not always possible to match recordsize to the actual write i/o, especially when compression is enabled. Moreover, we have no information on actual pool/dataset config, so I must assume default recordsize (128K) – shodanshok Jan 30 at 21:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.